[TangerineSDR] Seeking your Review and Comment on Tangerine SBC Functional Specification

[Rob Wiesler]

Hi, Bill.

On Tue, Oct 8, 2019 at 1:56 PM Engelke, Bill <bill.engelke at ua.edu> wrote:
> Hello Rob - thanks for your most helpful reply - I will be incorporating your thoughts into the next version.  I have a few follow-up remarks:
>
> - I will discuss the Ring Buffer concept with Nathaniel.  It seems like it could be useful, it's just that it would not be practical with today's hardware to run it at the huge data rate originally envisioned.

That's my thought.  I think we might (at least briefly) explore the
idea of a RAM-based ringbuffer (I haven't even done the
back-of-the-envelope calculation to know if this is even remotely
feasible - the N2 has either 2G or 4G RAM).  Again, I don't know which
phase this belongs to (I'll let you guys figure it out).

> - Excellent idea to add a statement about making the system able to survive power cycles. That was one of the things that made me abandon Linux for a number of years after it first became available - if the power went off, the system would often get bricked. I have been delighted to find that this is not the case with more modern implementations (i.e., Raspberry, Odroid N2 etc)

The difference is a matter of journaling file systems, better
per-program behavior, significantly better sync(1) semantics, and
per-package work towards idempotent software updates.  The last bit
remaining for TangerineSDR is atomic updates.

> - Discoverability: I am not sure of the value of this either, it is an artifact from how the OpenHPSDR-compatible systems work. With TangerineSDR, we do plan to support the config where there are multiple units on the same network (institutional setting); but the plan is that these would be individually configured using a web running in the SBC.

That plan sounds good.

> - Operating system: you mention Debian several times.  The Odroid N2 uses Ubuntu. If someone wants to go through the effort to gen up Debian to run on it, they are welcome to do so, but I would consider that out of scope for Phase 1.

They likely ship Ubuntu only to reduce the complexity of choice.
Debian runs perfectly well (I've seen a lot of mail to this effect
pass through my inbox).  Since we will want to generate our own image
and not simply use theirs, and they likely have no significant
Ubuntu-related modifications we want to keep, Ubuntu and Debian are
more-or-less fungible here.

In any case, this shouldn't be a sticking point.

> - Waiting for files to close: what I mean by this is: if a file is actively being uploaded to the server, the server should not attempt to open and read that file before the transfer is complete; in fact, I would expect the file transfer process to maintain exclusive rights to the file until transfer is done (and file closed).  What I have been imagining for Use Case 3 (data pre-processed by GNURadio in TangerineSDR) is that the FFT snapshots (of small chunks of bandwidth around WWV) would be uploaded in real time and stored as database records (rather than files), which makes them immediately available for real-time analysis on the server.  (This is not to say we can't also support Use Case 1, which involves uploading large files from the ring buffer - in which case we have the condition of waiting for a file transfer to complete before we can access the file on the server).  I'm sure there are other ways to "skin this cat," - if I am missing something, please advise.

Well, the classic way to do this is to process the uploaded stream as
it's being uploaded, if real-time processing is desired.  In that
case, the part that's doing the processing likely won't be reading the
data from the filesystem (or database) - the server that's accepting
the data will stream via memory as it arrives.  If chunks are
sufficiently small, then we can reduce the complexity of the system
(and keep (near-)real-time processing as an option) by using an
inotify-based watch loop, which can be set up to only trigger when the
file is completely written to the filesystem (I use this kind of thing
a lot in my own systems).

If we store the data in a database as the hand-off point between the
server and the real-time processing stage, we run into the problem of
notifying the processing stage when there's data to be found in the
database (sampling for new data is not performant).  There are
multiple easy solutions to this problem, but they're all at least as
hard as just passing the data to the processing stage via a pipe or
some other IPC method.  In contrast, if we write to individual files
on a real filesystem, inotify handles the handoff of "ownership".

> - Yes, mender.io is merely an example. I will look into how atomic updates are done with Ubuntu.

I'm not sure too many people have actually solved this problem the way
I think we need to, but proof to the contrary is very welcome.
Anything relevant to Ubuntu should be trivially transferable to Debian
(and vice-versa), *unless* it involves Canonical (in which case we'd
probably have to pay for it anyway).

> QUESTION: I am guessing that some geeks (including self here) will prefer to have control over when updates are applied, and others are happy to let the system handle these automatically; we probably should let people have the choice, shouldn't we? Maybe default to automatic, but let the user opt out of that?  Or, what do you suggest?

My thought is that we should ship with automatic updates enabled, but
ask the user what they prefer during setup (when they enter the upload
identity token and configure network access).  Then we just make sure
that updates don't happen while the system is being configured (easy).
That way we have a system that does the right thing out of the box,
but ultimately does whatever the user wants it to do, all without
generating some setting the user needs to hunt around for.

> Please keep me informed of your progress on building an example system.  We are being asked (by Nathaniel, et al) to make data collection methods common between TangerineSDR and the CWRU system (does it have a name?) ... I have been playing with a development version of the SatNOGS Network, and plan to use a modified version of it for our Phase 1 Central Control System & database.

We have a meeting tomorrow morning.  I'll keep you posted, and
probably introduce you to the guy who seems to be riding herd on the
students who are already working on that.

As far as a name goes, we've been calling it (or at least the radio
board that John Gibbons (N8OBJ) designed and built) the "Standards
Station Receiver".  This isn't exactly an unambiguous name, nor does
it properly refer to the entire system as a whole, so let's not
engrave that anywhere important just yet :) .

--
73 de AC8YV