[TangerineSDR] Seeking your Review and Comment on Tangerine SBC Functional Specification

[Rob Wiesler]

On Wed, Oct 9, 2019 at 3:08 PM Engelke, Bill <bill.engelke at ua.edu> wrote:
> A couple of data points for you -
>
> RAM-based ring buffer: Digital RF includes this as a standard utility, which I have tested and it works great. I have configured it on the Odroid and pushed data to it at a pretty high rate, and it got all the stuff written to separate spinning hard drive (at least right up until the Odroid's CPUs started bumping 100% utilization, which was 12 to 14 channels of I/Q data at 192 ksps). If you want to play with Digital RF, it is available at https://github.com/MITHaystack/digital_rf . Note: you must install HDF5 first.
>
> Regarding handling uploaded data - you raise a lot of good points about data storage and performance implications... I need to study on that a bit more.  At the very least, I envision that the storage method for I & Q data may want to differ from that of pre-processed data, and maybe between the ring buffer and firehose; but we can discuss. I will read up on inotify.

Somehow I thought you were talking about server-side stuff.  If we're
talking about storing data and forwarding data as independent steps,
inotify will still play part of the solution, but then if we need to
upload a backlog of files, we'll need something else to let us
enumerate files to upload without including any files that are
currently being written to.  In my experience, it's best to write to a
file on the same filesystem (maybe in the same directory, but with
some prefix and/or suffix that excludes it from the list of files to
upload or process (I often just prefix it with a '.', which nicely
omits it from shell glob expressions by default - Firefox uses a
".part" suffix).  Once it's completely written, renaming it (to remove
the prefix/suffix) is an atomic operation, and will trigger inotify.

Here's a simple (and somewhat fragile, in case you have weird
filenames) example of using inotifywait (a simple wrapper around
inotify) in a shell script to do something whenever a file is
completely written or moved to (or renamed within) the watched
directory:

    #!/bin/sh
    inotifywait -m . -e close_write -e moved_to 2>/dev/null |
    while read path action file; do
        process_file "$path$file"
    done

The inotify API itself is brutally simple, but if you want assistance,
let me know.

--
73 de AC8YV