[TangerineSDR] Filename structure, Node info contents, other stuff to ponder

Wed May 6 14:46:03 EDT 2020

Hi John,

Two thoughts:

  1.  While I think the filename is important, I think it should be more for convenience than for actual storage of real information. That is, the file itself should also contain all of the data/metadata to recreate the filename. It is the data inside of the file that should be considered the definitive record.
  2.  Regarding node numbers, perhaps an alpha-numeric approach is better, than just numeric? It gives a lot more flexibility in the naming space.

73 de Nathaniel W2NAF

From: TangerineSDR <tangerinesdr-bounces at lists.tapr.org> On Behalf Of John Gibbons via TangerineSDR
Sent: Wednesday, May 6, 2020 11:54 AM
To: TAPR TangerineSDR Modular Software Defined Radio <tangerinesdr at lists.tapr.org>
Cc: John Gibbons <jcg66 at case.edu>; David Kazdan <dxk10 at cwru.edu>
Subject: Re: [TangerineSDR] Filename structure, Node info contents, other stuff to ponder

Rob,

Any node can be collecting multiple files - that's what the TYP and MOD fields are for.  When finished, the LC PSWS will be collecting 6 files (4 FREQ [with freq modifier added] data sets, 1 MAG and 1 TMP) per day per node for any given node per given UTC day.

If data collection is interrupted, the system just continues adding data for that UTC day in the already existing file for that data set [i.e. appends and does not start over].
That was one of the major mods I did to FLDIGI to accommodate this situation.
Collecting the exact same data set on two different machines at the same node on the same UTC day makes no sense and is not allowed.  A node will only be allowed to submit one file per specific data set per UTC day.
There will be no merging of files upstream.

As far as the indication of UTC day (using the ISO format) in the filename, I'm already doing that and there is no simpler way to indicate this and anyone using the system already understands this.  Adding anything here just makes the filename longer.... and it's not necessary for the developers.
As I've stated in the spec, data sets are based on UTC day and labeled (in ISO format) as such.  It doesn't get much simpler than this.

The file naming scheme (using the ISO date) is intended to help someone who knows what they are doing to visually determine the UTC day's contents of the file (per Nathaniel's request).  I believe I've achieved this as simply as possible.

Semantics about user root and user pi are why I did the system call - someone may choose a different username and I've already accommodated this situation.  And it does work (David Kazdan tested this).
And any user should NOT be running as root - breaks all the protections and is bad practice in general.  Nuf' said.

Using a system call in PYTHON to get the ENV variable for the home directory is an acceptable practice to determine this.  We're going to be setting up the systems, so this is not going to be a problem long term.

We have to be able to have our files usable on WIN7/10 systems, so the use of special characters in the filename that linux can handle is simply not allowed.  We have to be compatible.

And now that I think about it, the - [minus] sign IS usable in WIN7/10, so I'll fix that in the spec.  (And I'm using it in the filename - OOPS!)

John N8OBJ

John C. Gibbons
Director - Sears Undergraduate Design Laboratory
Dept. of Electrical Engineering and Computer Science
Case Western Reserve University
10900 Euclid Ave, Glennan 314
Cleveland, Ohio  44106-7071
Phone (216) 368-2816<tel:216-368-2816> FAX (216) 368-6888<tel:216-368-6888>
E-mail: jcg66 at case.edu<mailto:jcg66 at case.edu>

On Tue, May 5, 2020 at 11:50 PM Rob Wiesler via TangerineSDR <tangerinesdr at lists.tapr.org<mailto:tangerinesdr at lists.tapr.org>> wrote:
On Tue, May 05, 2020 at 23:01:15 -0400, Phil Erickson via TangerineSDR wrote:
> On Tue, May 5, 2020 at 10:05 PM John Gibbons via TangerineSDR <tangerinesdr at lists.tapr.org<mailto:tangerinesdr at lists.tapr.org>> wrote:
>> On Tue, May 5, 2020 at 8:38 PM Rob Wiesler via TangerineSDR <tangerinesdr at lists.tapr.org<mailto:tangerinesdr at lists.tapr.org>> wrote:
>>> (David is probably on the TangerineSDR list, but I don't know that for
>>> sure, so he may get an extra copy of this message (sorry).)

David is *not* on the list.  That's good to know.  Maybe I should stop
my habit of trimming the Cc header so aggressively for this list.

>>> Does the date in the filename refer to:
>>> - the beginning of the record, or
>>> - the end of the record, or
>>> - a single day in its entirety, or
>>> - at most a single day, but no (significant) part of any other day?
>
>   Your files only have the YYYY-MM-DD form of the ISO date in them.  This
> seems to imply that there can be only one file per day.  What if the
> instrument dies for a while and then restarts on the same day?  I guess I'm
> wondering why you wouldn't use the fully qualified ISO (e.g.
> 2020-05-05T03:00:00).  Maybe I'm missing something obvious.

This hasn't been answered yet exactly, but the current low-cost PSWS
fldigi scheme rotates the file at the end of the UTC day, so there's a
one-to-one correspondence between UTC day and filename.  This is
entirely unclear from the filename specification, though.

In any case, I find it entirely too likely that we'll see two different
files from nonoverlapping times with the same name from the same node.
Imagine that someone sets up a system, starts it logging, forgets the
root password an hour later, and sets it back up again.  Are we going to
merge the two files upstream?  I would rather not have to.
Misconfigured systems (for instance, cloned systems containing duplicate
node IDs) are something we have to look out for, but this is a case of a
correctly-configured system that just happens to have lost its memory.

Using the full ISO date (minus the (to us) useless time zone offset
portion) corresponding to the time of the first record in the file would
fix this problem.

>>> Let's at least specify WWVdata/ as existing relative to "the user"'s
>>> home directory, instead of /home/pi (it can't hurt to be explicit).
>>> Also, you have a typo, where you say that "~" is the "root filesystem
>>> for user", which is not a thing (you mean "home directory").
>>
>> For the RasPi OS (Linux), I use a system call to define ~ which is the
>> BASE of the user's directory structure (root was a BAD choice here - not
>> intended to confuse it with root user)

It would be even worse to get it confused with the real root directory,
which is "/".  Or with "user root"'s directory, which is typically
"/root" (pronounced "slash-root" to disambiguate).

Don't call it a "base", please.  This one one of those cases where
there's one right term with no competitors.  Just call it the user's
"home directory".

For today's dose of useless pedantry from Rob, ~ is expanded to the real
path of the user's home directory by a globbing function, based on the
value of the HOME environment variable.  Since environment variables are
loaded into the process's memory space literally before they get a
chance to start, this doesn't involve a system call at all (system calls
are a way for userspace processes to invoke functions provided by the
kernel, not a generic term for functions provided by the system at
large).

>>> Is there a reason you're avoiding a second '.' in the filename?  It's a
>>> little awkward to use 2p5 for 2.5, and that second period isn't going to
>>> confuse any software or upend the sorting.
>>
>> I stayed away from . and - in the filename as Windows users would get into
>> trouble here (I stuck to just the _ char) and I think we have to cater to
>> that limitation of Windows 7/10/whatever.

Only when renaming files from the GUI does this become difficult on
Windows, and in any case, we don't want users doing that.  Not exactly a
feature, but not a bug, either.

>>>> I have the original .doc that created this - let me how we should handle
>>>> version control from here.  (Nathaniel?)
>>>
>>> Please turn this into a plain text file.  I can read PDFs, but it's not
>>> ideal, and I'm getting sick and tired of specification documents in
>>> other non-textual formats.  A plain text specification file has these
>>> properties:
>>>
>>> - Small file size (because this is 2020 and it totally matters)
>>> - Less wasted visual space when the document isn't all that long (again,
>>>   wishlist-grade)
>>> - Universally readable
>>> - Universally modifiable by the recipient (so recipients can offer
>>>   suggestions formatted as a pull request)
>>> - Diffs between revisions can be generated trivially, so that recipients
>>>   can:
>>>   - offer suggestions formatted as a patch
>>>   - figure out what changed without scanning the entire document for
>>>     thin red/yellow lines on a white background (very important to me)
>>> - Mergeable when put in version control (in addition to all the other
>>>   properties above you would want for version-controlled documents)
>>
>> This will be available online when we get it into version control.

So long as it's not a PDF in version control, please.  The only thing
sadder than a sourceless binary in version control is one that also has
the wrong permission bits set.

--
TangerineSDR mailing list
TangerineSDR at lists.tapr.org<mailto:TangerineSDR at lists.tapr.org>
http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20200506/fbd9ef36/attachment-0001.html>