[TangerineSDR] Fwd: Fwd: magnetometer data file

Mon Jan 18 15:11:15 EST 2021

---------- Forwarded message ---------
From: David Witten <wittend at wwrinc.com>
Date: Mon, Jan 18, 2021 at 10:42 AM
Subject: Re: Fwd: magnetometer data file
To: John Gibbons <jcg66 at case.edu>, Dr. Nathaniel A. Frissell Ph.D. <
nathaniel.frissell at scranton.edu>
Cc: Julius Madey <hillfox at fairpoint.net>, Dave Larsen <kv0s.dave at gmail.com>

John, et al,

Part of the 'problem' here is that we do not have uniform terminology, and
also that we are discussing two different programs that I'm writing.
None of this impacts the design of the current Grape project.

I have one program which creates an executable called 'runMag' that is the
original testing code.
The other program is 'runRM3100' and makes an effort to follow the Grape
project metadata and file storage scheme.
Both are command line only and are written in pretty portable C.
They are meant to be straightforward, and I deliberately have avoided any
cleverness.

The runMag program  makes no pretense of storing significant metadata and
uses a different file name format for the saved data.
This is what Jules, Dave Larsen, Hyomin, and I are using to capture output
'logs'.
I am making very few changes to this at the present time, because I'm busy
elsewhere.
But I did push a version 0.0.15 to GitHub last night to try to clean up the
CSV output format somewhat for Jules and others.
The JSON code definitely needs to be re-tested.
And i might have even introduced an error in the way the data is adjusted
for averaging in the ASIC device.
This needs to be checked, but I don't have time today.

The runRM3100 program is almost done, but has a number of 'fiddly' aspects
that take time to implement and test.

LATER, both programs would benefit from some rework on the i2c code and on
the way that measurement interval is implemented.
In particular, jitter in the measurement interval needs to be investigated
under high CPU load.

Metadata in this context seems to refer to a number of different things:

   - The information that is concatenated into the filename used for
   storing the daily files that capture measurements. This is different
   between the two programs, but not wildly incompatible.
   - The layout of a group of folders or directories that hold all of the
   data related to the Grape project.
      - These folders contain input to the program to direct its operation.
      - They also contain information about a specific site or node that is
      incorporated into the header of the output.
      - They also include work space where data is collected and stored.
   - The layout of header info and data within the files that contain the
   collected values.

To respond again to Jules's original request, all of these issues in the
runMag program either have been or will be addressed promptly, however you
wish them to be done.

All of the work on the runRM3100 is done with the intent that it will
maximize compatibility with the current and future Grape architectures.
All of this work is done with full knowledge that most of these matters
will be mooted by the use of a Digital RF format for all output from the
project.  Eventually.

Dave Witten, KD0EAG

On Sun, Jan 17, 2021 at 11:53 PM John Gibbons <jcg66 at case.edu> wrote:

> David & Jules,
>
> Formats are NOT set in stone - if this msg is in reference to the column
> headers I sent you as a suggestion then you misinterpreted what I was
> saying.
>
> You set the format of the data up to what the experiment produces and
> needs.  YOU control that.  Nathaniel wants the full ISO timestamp, but
> other than that you do what you want.
>
> The metadata that is attached to the head of each day's Grape data file is
> for Bill E. to be able to read in as a standard format.  This also may
> change if Bill needs something else.
> It also follows along with the data file so it can be identified easier as
> a standard format as to what the data is from.
>
> For Bill's Tangerine files, he doesn't even use this - he is using a
> Digital RF format.
>
> If you elaborate on your problem maybe I can be of more assistance.
>
> John N8OBJ
>
>
>
> John C. Gibbons
> Director - Sears Undergraduate Design Laboratory
> Dept. of Electrical Engineering and Computer Science
> Case Western Reserve University
> 10900 Euclid Ave, Glennan 314
> Cleveland, Ohio  44106-7071
> Phone (216) 368-2816 <216-368-2816> FAX (216) 368-6888 <216-368-6888>
> E-mail: jcg66 at case.edu
>
>
>
> On Sun, Jan 17, 2021 at 9:18 PM David Witten <wittend at wwrinc.com> wrote:
>
>> Jules,
>>
>> No, it is mostly set in stone.  Trouble is that no one ever asked us (or
>> anyone else, as far as I know).
>>
>> Dave
>>
>> On Sun, Jan 17, 2021 at 7:48 PM Julius Madey <hillfox at fairpoint.net>
>> wrote:
>>
>>> Hmm, if the metadata describes the data format in the file, then, in
>>> theory, different formats are allowed.?????
>>>
>>>
>>> On 1/17/2021 7:19 PM, David Witten wrote:
>>>
>>> Jules,
>>>
>>> I entirely understand.
>>>
>>> I have a patch from Dave that apparently addresses most of those
>>> things.  I've tried to adjust the averages, but I'm not sure.   I made most
>>> of these changes to the newer program, but I'm, still working through
>>> creating and assembling all to the metadata there.  I've kind of lost the
>>> plot for the moment.
>>>
>>> But try version 0.0.15 that I've just committed to github.
>>>
>>> Dave Witten
>>>
>>>
>>> ---------- Forwarded message ---------
>>> From: Dave Larsen <kv0s.dave at gmail.com>
>>> Date: Sun, Jan 17, 2021 at 5:22 PM
>>> Subject: Re: magnetometer data file
>>> To: David Witten <wittend at wwrinc.com>
>>>
>>>
>>> Dave
>>>
>>> They address many of his issues an make data analysis easier.
>>>
>>> I did not address averages other than your quadratic mean.  (Total)
>>>
>>> I changes are only in the CSV switch.
>>>
>>> I did not try to find all places I needed to change.
>>>
>>> Let know if you have specific questions.
>>>
>>> Dave
>>>
>>>
>>>
>>>
>>> On Sun, Jan 17, 2021, 5:01 PM David Witten <wittend at wwrinc.com> wrote:
>>>
>>>> Dave,
>>>>
>>>> Does your  'patch' accomplish these things?
>>>> My mind has been entirely on other things than runMag.
>>>> If it does, I'll make the shift and push it to Github.
>>>>
>>>> Dave
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Julius Madey <hillfox at fairpoint.net>
>>>> Date: Sun, Jan 17, 2021 at 4:38 PM
>>>> Subject: magnetometer data file
>>>> To: John Gibbons <jcg66 at case.edu>, David Witten <wittend at wwrinc.com>
>>>>
>>>>
>>>> Help me out here guys.
>>>> The version of RunMag I'm using includes a text label for each
>>>> component:
>>>> time:
>>>> rTemp:
>>>> lTemp:
>>>> x:
>>>> y:
>>>> z:
>>>> rx:
>>>> ry:
>>>> rz:
>>>>
>>>> While that format is very handy for getting use to the magnetometer
>>>> operation and general experiments, what about reducing the file size by
>>>> using the same order but with only comma separated values for the working
>>>> data repository.
>>>>
>>>> And, either use the calculated x,y,z values from known cycle count and
>>>> average count
>>>> or
>>>> only the raw values with cycle count and average count in the metadata
>>>> for the file
>>>>
>>>> Any thoughts?
>>>>
>>>> Thanks,
>>>> 73,
>>>> Jules
>>>>
>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20210118/15317b87/attachment.html>