[TangerineSDR] Uploading of spectrum data from TangerineSDR to Central Control

[Gerry Creager - NOAA Affiliate]

I got caught in a Cray continuum event horizon and fell off email for
awhile. Probably phone, or if there's enough interest, Zoom. When's good?

gerry

On Tue, Jun 9, 2020 at 6:25 PM Engelke, Bill via TangerineSDR <
tangerinesdr at lists.tapr.org> wrote:

> Gerry, I replied to you using the email address you have in qrz.com – 73-
> Bill
>
>
>
> *From:* TangerineSDR <tangerinesdr-bounces at lists.tapr.org> *On Behalf Of *Gerry
> Creager - NOAA Affiliate via TangerineSDR
> *Sent:* Tuesday, June 9, 2020 1:14 PM
> *To:* TAPR TangerineSDR Modular Software Defined Radio <
> tangerinesdr at lists.tapr.org>
> *Cc:* Gerry Creager - NOAA Affiliate <gerry.creager at noaa.gov>
> *Subject:* Re: [TangerineSDR] Uploading of spectrum data from
> TangerineSDR to Central Control
>
>
>
> Bill,
>
>
>
> lftp, sftp, rsync, etc.  are viable alternatives. As usual, in my world,
> there's something that might work for sending data and allowing its request
> by multiple hosts, as well. It's used to push weather data all over the
> place and is widely used world-wide. Let's talk offline a little and let me
> get a slightly better sense of the data you're moving and if the LDM system
> might be useful or might just be, to borrow Phil's comment in another
> thread, a "hammer looking for a nail".
>
>
>
> 73
>
> Gerry N5JXS
>
>
>
>
>
> On Tue, Jun 9, 2020 at 4:38 PM Engelke, Bill via TangerineSDR <
> tangerinesdr at lists.tapr.org> wrote:
>
> Hello All:
>
>
>
> I have a prototype of a process working for moving (raw) spectrum data
> from the TangerineSDR to the Central Host; I would like any interested
> colleagues to review and comment on it, since it is such an important piece
> of the system. I have a mockup running in Ubuntu on a virtual server, but
> the final Central Host process will run under CentOS and Django. Here’s how
> the process works:
>
>
>
>    1. When in Ringbuffer mode, the TangerineSDR collects one to 16 bands
>    of spectrum data, saving it in a ring buffer in Digital RF format. (Typical
>    actual use would probably be the ~ 5 bands of WWV).
>    2. Every so often (say, 120 seconds), the TangerineSDR sends a
>    heartbeat packet to the Central Host.
>    3. If the Central Host has been given a data collection command (by a
>    superuser), it will reply to the heartbeat with a Data Request that has a
>    start and end time.
>    4. The TangerineSDR finds all the files and metadata collected during
>    the requested time frame and uses tar to put into a single file (optionally
>    compressed).
>    5. The upload file gets named according to the (still being refined)
>    CWRU naming convention (except that band info will not appear in the file
>    name since a DRF file will usually contain multiple bands).
>    6. The TangerineSDR optionally sets the maximum upload rate
>    (“throttle”) and uses the utility lftp to send the file to the sftp server
>    running on the Central Host. Each user has their own “incoming” directory.
>    7. The Central Host checks the incoming directories at certain
>    intervals; when it finds a file package, it moves it to a central
>    collection and puts a cross-reference into the relational database.
>
>
>
> Notes: I thought about a number of different ways of moving the data from
> TangerineSDR to the Central Host; this way seems promising because it
> offers the following benefits –
>
>
>
>    - lftp is restartable in case the upload is interrupted. Some uploads
>    will run for a long time.
>    - Using sftp as the protocol protects the data in transit.
>    - The Central Host can handle a lot of uploads at once without
>    burdening the website.
>    - Since each user has a unique UID and password, and data travels
>    encrypted, we should be able to minimize hackers uploading garbage.
>    - The TangerineSDR has another mode (“snapshotter”) which
>    pre-processes raw spectrum into frequency domain data – this will be
>    handled differently (more to come on that).
>
>
>
> I’m sure I have not thought of everything though;  if you have a
> suggestion on how to do this better, it is most welcome – thanks for your
> help!   -73- Bill  AB4EJ
>
> --
> TangerineSDR mailing list
> TangerineSDR at lists.tapr.org
> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>
>
>
>
> --
>
> Gerry Creager
>
> NSSL/CIMMS
>
> (C) 979.229.5301 <--- NOTE THAT MY OFFICE NUMBER HAS CHANGED
>
> ++++++++++++++++++++++
>
> *The way to get started is to quit talking and begin doing.*
>
> *   Walt Disney*
> --
> TangerineSDR mailing list
> TangerineSDR at lists.tapr.org
> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>


-- 
Gerry Creager
NSSL/CIMMS
(C) 979.229.5301 <--- NOTE THAT MY OFFICE NUMBER HAS CHANGED
++++++++++++++++++++++
*The way to get started is to quit talking and begin doing.*

*   Walt Disney*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20200611/d5437ca5/attachment-0001.html>