[aprssig] High speed datbase inserts (CGI)

[Ken Linder]

Andrew,
I might not be a findu expert but I manage my share of data (> 3 TB) at
work.  I would say that it would probably be easier to just shove the data
into mySQL.

UNLESS...  Your mySQL server is quite underpowered.  Then, it might be
better to use the temporary files and let the mySQL engine 'breate'.

My 2 cents worth.
73's, Ken
KC7RAD

----- Original Message ----- 
From: "Andrew Rich" <vk4tec at tech-software.net>
To: "TAPR APRS Special Interest Group" <aprssig at lists.tapr.org>
Sent: Monday, February 28, 2005 10:35 PM
Subject: [aprssig] High speed datbase inserts (CGI)


> Howdy,
>
> I have a situation at my work that is very close to APRS.
>
> I was wondering if someone could pass on some words of wisdom.
>
> I have a stream from a data source not unlike and APRS server stream.
>
> I would like this to be split up and end up in a MySQL database to be
> searched on.
>
> I have been experimenting with writting to files every 10 minutes and then
> doing inserts into mysql
> from these files. With the idea of giving the database time to breath for
> searches.
>
> I have some questions regarding perl. In the duration I am closing off one
> file and opening up another,
> would the telnet data be "pooled" by Net/Telnet.pm.
>
> Would I be better off inserting straight into the database ? and not worry
> about files ?
>
> How does findu.com do it ?
>
> I think I see two ways of doing this
>
> 1) Write data from the telnet stream into files each minute, then start a
> new file and process the old files
> 2) Place data straight into the mysql database
>
> My only concern is bottle necks.
>
> Comments ?
>
> Andy VK4TEC