[TangerineSDR] Updated Detailed Design Specification for PSWS Local Host (v0.1)

[Rob Wiesler]

On Fri, Nov 15, 2019 at 22:25:00 +0000, Engelke, Bill via TangerineSDR wrote:
> To all - Attached please find 2 files.  This is the latest update
> (Version 0.1) of the PSWS Local Host (SBC) Detailed Design
> Specification, both with all markup and no markup.

Thanks, Bill.

> I tried to create a version with "simple markup" where it just puts a
> red line next to anything that was changed; but MS Word will not
> produce a pdf with that markup style.  Strange. I tried all kinds of
> ways.

I actually quite like the existing style.  It makes it quite obvious
what I have to pay attention to.

> There is a lot of new stuff in this version, you can see a summary of
> the changes on p. 2.   Please review and comment - thanks -

I have a lot of comments:

- Remember, to achieve our goals relative to software distribution,
  maintenance, updates, and installation, we basically cannot use HardKernel's
  Ubuntu image as-is, and we definitely cannot pile a heap of manual builds of
  unpackaged software on top of that.

  - Security updates are not something we can leave to HardKernel, even for the
    subset of packages they modified for the image.

  - We want the simplest possible installation.  That means everything must
    come preinstalled, or modification of an existing Debian-based operating
    system must be limited to the addition of a sources.list.d entry and
    an archive keyring, and subsequent installation of packages (which
    is scriptable).

  - The HardKernel image uses an ext4 (I believe) root filesystem.  Since we
    want atomic security updates and minimal downtime (and all in-tree
    filesystem drivers), we'll need to convert that to a btrfs filesystem (and
    move /boot to vfat, as Das U-Boot isn't that great with btrfs just yet).

  - The HardKernel distribution is aging.  Fortunately, part of why they
    haven't kept it up-to-date is because they've been upstreaming their kernel
    changes, i.e. the only reason their disk image even exists is because
    officially-distributed kernels from that point in time needed patching
    because their changes did not get backported upstream (because that's how
    stable distributions work).  I'm still verifying this, so take it with a
    grain of salt.

  - We want as much of this as possible to run on a desktop in *exactly* the
    same form as it does on the LH, for lots of reasons.

  - Obviously, Phase 1 will probably still use the HardKernel image as a base,
    but it will have to be modified to convert partition 2 to btrfs, move /boot
    to partition 1, configure our APT repo, and install our packages by default.

  - gnuradio is already packaged for Debian/Ubuntu.  If there's a specific
    reason why that version (or the version in backports) is unsuitable,
    I don't remember anybody mentioning why.  Additionally, if it
    actually is unsuitable, we should instead distribute a modified
    version of the package via our repo instead of building from source
    on each end device.

  - I now plan to package digital-rf for Debian.  It looks to be in relatively
    good shape, and the dependencies all appear to be in Debian already.  Any
    package I create can live in our repo (which doesn't yet exist) until it
    makes it to Ubuntu.

  - The RedPitaya out-of-tree module should also make it into a real package.

- Getting the time via NTP is not a network operation if a GPSDO is
  installed (the diagram is misleading on this point).  NTP will still
  be used, but it will act as a stratum 1 time server, with its inputs
  coming from the GPSDO.  Additionally, modern computers do not udpate
  their system clocks in the way you seem to think - the requirement
  that "the LH system clock should be updated no more frequently than
  once every 15 minutes" is bogus and harmful.  Modern system clocks are
  slewed continuously, not stepped, and this is for the benefit of WSPR,
  not its detriment.  Again, this is something to let NTP handle and not
  think too hard about.

- The FT8 decoder should be able to decode our samples without needing them to
  be written to a disk.  It's conceivable that the FT8 decoder is not
  sophisticated enough to read from stdin or a pipe, but in that case we still
  need to save the sample to a file in tmpfs (RAM disk) instead of involving
  flash memory.

- I believe that callsign monitoring is totally something we should use
  a local event bus for, and MQTT is perfect for that (and fot the other
  event types you mention that might generate email).  That means that
  each observed callsign is published locally (and probably discarded
  due to lack of subscriptions), and our event-to-email process
  subscribes to these events.  I don't see a limit of 8 callsigns as
  being either reasonable or useful - the system should be designed to
  listen for as many callsigns as the user wants.  There shouldn't be
  any technical reason against this, and any interim implementation that
  imposes a cap will necessarily be more complicated.

- A comment in the callsign-to-email section mentions that Google Mail
  et. al. no longer accepts "this kind of email".  That is not
  necessarily true.  After having spent way too long last week
  configuring my own mail server and offline IMAP storage for my
  existing Gmail account (and not too long configuring a lab-bound RasPi
  to send mail for Kristina Collins (KD8OXT)), I can say confidently
  that our options and constraints are these:

  - The listed corporate email providers don't like mail sent without a DKIM
    signature.  That's basically the easiest part of setting up a mail server
    (including picking which packages to install), but it means that mail
    should be sent through a private SMTP relay (such as Gmail's servers) that
    performs this task, because we're never going to get DKIM working on
    systems in the field.

  - Providing credentials (username and password) for a correctly-configured
    SMTP account are sufficient.  If you have a Gmail account, your normal
    login will work (and you can make a new Gmail account trivially).
    Receiving mail from a Gmail account, on the other hand, is a complete
    nightmare, but thankfully we don't have to deal with that.

  - Python's smtplib module is enough for this task.  I believe it's part of
    the standard Python distribution.  Sending an email using it takes about a
    dozen lines of code (example on request, but the online
    documentation isn't too bad).

- The software update mechanism is not present in the LH System Architecture
  diagram.  The additions I would expect to see look like this:

  ┏━━━━━━━━━┓
  ┃ MAINCTL ┃
  ┗━━━━━━━━━┛
      ↓ ↑
  ┌───────────────────────────────────────────────────┐   ┌───────────────────────┐
  │ Local apt/dpkg/apt-get/aptitude/etc. (Unmodified) │ ← │ TangerineSDR APT repo │
  └───────────────────────────────────────────────────┘   └───────────────────────┘
    ↑        ↑
    ↑        TangerineSDR sources.list.d entry, TangerineSDR archive keyring
    ↑
    Hook scripts for before/after package installation (btrfs snapshot management)
    ↑
    initrd hook script to recover from incomplete updates by reverting to
    last btrfs snapshot (i.e. runs from initramfs before real rootfs is
    mounted read-write and init (systemd) runs)

- You mention in the "Remote System Updates" section, "Need to select a package
  for enabling this."  What do you mean by that?

- Timestamps are still listed as being 128-bit.  I believe Ryan gave a good
  argument as to why this is too many.  Even if 64 is too few (and it probably
  isn't), 128 is definitely overkill.  That's not to say that this is
  not acceptable - I'm just trying to point out something that appears
  not to be part of what I understand as our consensus.

--
73 de AC8YV