[TangerineSDR] VLF-RX tools

[Phil Erickson]

Hi Jonathan,

  As Ryan said, some kind of a tool as he described it which reads from a
pipe should be straightforward.  The example you've given is a way to get
started, and I'll talk with him later about getting that implemented.  It's
a use case we want to support.

Cheers
Phil W1PJE

On Thu, Mar 11, 2021 at 8:16 PM Jonathan <emuman100 at gmail.com> wrote:

> Hi Phil,
>
> I understand now. The same issue has been encountered in seismology with
> the many different formats to store seismic wave data. The current winners
> now seem to be SAQ and miniseed formats, where miniseed has the ability to
> store metadata, such as instrument type and calibration data required for
> that seismic instrument. Both formats and others have detailed spec
> documentation. (I believe you can store metadata in the SAQ format as
> well, but I'm not sure)
>
> In that case, since purely real streams can be supported, is there a
> possibility of a conversion tool becoming a reality? I cannot support
> writing a conversion tool, but others seem to want such a utility. If you
> go to the root of the vlfrx-tools site and navigate to "live streams",
> then on any one of the streams, click "events", a page will come up that
> shows whistler and chorus events (with a lot of false triggers due to
> motors starting up). With each event, you can download the .vt file and
> use it as an example file for a conversion utility.
>
> Jonathan
> KC3EEY
>
>
> On Thu, 11 Mar 2021, Phil Erickson wrote:
>
> > Date: Thu, 11 Mar 2021 19:57:54 -0500
> > From: Phil Erickson <phil.erickson at gmail.com>
> > To: Jonathan <emuman100 at gmail.com>
> > Cc: TAPR TangerineSDR Modular Software Defined Radio
> >     <tangerinesdr at lists.tapr.org>
> > Subject: Re: [TangerineSDR] VLF-RX tools
> >
> > Hi Jonathan,
> >
> >  There are only a very few formats that are designed for long term and
> > cross-platform "survival" across many different operating systems and
> > languages, and which have enough embedded metadata to describe the data
> in
> > enough detail to allow applications that were not thought of at the time
> of
> > recording.  HDF5 is the product of a huge amount of work - decades of
> > person-years - to enable that sort of survivability.  We therefore use it
> > as the carrier, with DigitalRF as the particular data structure
> implemented
> > in it.
> >
> >  I think Ryan's separate message is a workflow that would allow all the
> > good toolsets to be implemented without changing the way that the
> recording
> > happens.  Note as well that we do support purely real streams (no need to
> > always have I and Q) and getting a good use case which implements that is
> > on our list of ways we want to move forward with the format.
> >
> > Cheers
> > Phil W1PJE
> >
> > On Thu, Mar 11, 2021 at 7:47 PM Jonathan <emuman100 at gmail.com> wrote:
> >
> >> Hi Phil,
> >>
> >> When you mean "not always persistent across decades of measurements", do
> >> you mean sampling rate and sample length inconsistency across different
> >> data sources?
> >>
> >> Yes, it is a purely real signal. There is a utility called vtresample
> that
> >> will resample the VLF stream and maintain timestamp alignment. And you
> are
> >> exactly right, there is a file in the source called vtlib.h which looks
> to
> >> define the timestamping format. Many of the utilities have this as an
> >> #include.
> >>
> >> The author must not have felt a need to document the format. He's not
> >> affiliated with a university and I don't believe he's a licensed ham,
> >> however, he's an active VLF enthusiest. He must have written his tool
> set
> >> and used his format to maintain persistence in the data. I have always
> >> felt that his format was adequate, especially when using his stream
> >> manipulation utilities which align and maintain timestamping.
> >>
> >> I wasn't sure about the reason for the insistance on using DigitalRF for
> >> VLF data analysis and storage on the TangerineSDR other than that's what
> >> the grants are paying for everyone to support. It's such a useful tool
> set
> >> that it seems like a waste not to utilize it.
> >>
> >> Jonathan
> >> KC3EEY
> >>
> >> On Thu, 11 Mar 2021, Phil Erickson wrote:
> >>
> >>> Date: Thu, 11 Mar 2021 19:12:58 -0500
> >>> From: Phil Erickson <phil.erickson at gmail.com>
> >>> To: Jonathan <emuman100 at gmail.com>
> >>> Cc: TAPR TangerineSDR Modular Software Defined Radio
> >>>     <tangerinesdr at lists.tapr.org>
> >>> Subject: Re: [TangerineSDR] VLF-RX tools
> >>>
> >>> Hi Jonathan,
> >>>
> >>>  We'll take a look.  For a purely real signal (e.g. cos(\omega t)), of
> >>> course you would need exp(j \omega t) in I and exp(-j \omega t) in Q.
> I
> >>> believe DigitalRF may have the ability to capture purely real signals
> >>> without the need to do 2X expansion into I and Q, but I'll ask.  Things
> >>> like PCM audio streams are unfortunately not always persistent across
> >>> decades of measurements (we know this sort of thing from hard
> experience
> >> at
> >>> Haystack), so we choose HDF5 for long term persistence.
> >>>
> >>>  I don't have the time at the moment to wade through code, but if this
> is
> >>> C code, it is likely there is a header file (".h") somewhere that
> >> contains
> >>> the stream format.  You might post that here when it's found.  For this
> >>> application, "derived from the source code" definitely needs to be
> >> replaced
> >>> with a formal data specification and that could form the start of it.
> >>>
> >>> Cheers
> >>> Phil W1PJE
> >>>
> >>> On Thu, Mar 11, 2021 at 7:02 PM Jonathan <emuman100 at gmail.com> wrote:
> >>>
> >>>> Hi Phil,
> >>>>
> >>>> The stream format is not documented, however, it can be derived from
> the
> >>>> source code of many of the utility programs, like vtcard, vtcat,
> vtraw,
> >>>> and vtwavex to name a few. It's single-channel or multi-channel pcm
> with
> >>>> timestamping information. vtcat can combine single channel streams,
> say
> >>>> from an E-Field receiver and two B-Field receivers into a
> multi-channel
> >>>> stream and align timestamps of all channels.
> >>>>
> >>>> I'm not an expert at C by any means, and I haven't really looked at
> >>>> understanding the stream format in the source code yet. For a
> conversion
> >>>> utility, would the stream have to be copied and phase-shifted to
> >> generate
> >>>> IQ data?
> >>>>
> >>>> vlfrx-tools captures and works with data in its original form. From
> the
> >>>> soundcard, it's a pcm audio stream. There are utilities vtrtlsdr and
> >>>> vtsdriq that read data from a RTL dongle or SDR-IQ receiver
> >> respectively.
> >>>> vtsdriq can output either I and Q data in two channels or a single
> >> channel
> >>>> stream.
> >>>>
> >>>> Jonathan
> >>>> KC3EEY
> >>>>
> >>>> On Thu, 11 Mar 2021, Phil Erickson wrote:
> >>>>
> >>>>> Date: Thu, 11 Mar 2021 04:33:08 -0500
> >>>>> From: Phil Erickson <phil.erickson at gmail.com>
> >>>>> To: TAPR TangerineSDR Modular Software Defined Radio
> >>>>>     <tangerinesdr at lists.tapr.org>
> >>>>> Cc: Jonathan <emuman100 at gmail.com>
> >>>>> Subject: Re: [TangerineSDR] VLF-RX tools
> >>>>>
> >>>>> Hi Jonathan,
> >>>>>
> >>>>>  I looked at the documentation (HTML page) but I cannot find a
> precise
> >>>>> definition of the stream format.  Where is that located or could you
> >>>>> summarize that?  It would then be possible for those of us familiar
> >> with
> >>>>> DigitalRF, and in particular the Haystack team who maintains it, to
> >>>> examine
> >>>>> the ways in which a .vt file could be translated to DigitalRF or the
> >>>> other
> >>>>> way around.
> >>>>>
> >>>>> 73
> >>>>> Phil W1PJE
> >>>>>
> >>>>> On Wed, Mar 10, 2021 at 10:46 PM Jonathan via TangerineSDR <
> >>>>> tangerinesdr at lists.tapr.org> wrote:
> >>>>>
> >>>>>> Hi Tom,
> >>>>>>
> >>>>>> Thank you for looking at the documentation for vlfrx-tools! I'll
> list
> >>>> some
> >>>>>> useful utilities included in vlfrx-tools to try to illustrate how
> >> useful
> >>>>>> they can be. A utility can be written to convert to DigitalRF/HDF5,
> >>>>>> however, vlfrx-tools does not normally generate IQ data.
> >>>>>>
> >>>>>> For others on the list, the vlfrx-tools documentation is
> >>>>>> available at http://abelian.org/vlfrx-tools/ and this also includes
> >> the
> >>>>>> source code. Also, for the others on the list, I'd like to make the
> >>>>>> argument that vlfrx-tools is advantageous because of its useful
> >> toolset
> >>>> in
> >>>>>> analysis and post-processing of both natural radio, amateur radio in
> >> the
> >>>>>> VLF band, and VLF transmitters. Such tools don't exist (to my
> >> knowledge)
> >>>>>> for formats like DigitalRF. Since vlfrx-tools is such a diverse and
> >>>>>> usefull toolset that already exists, it's advantageous to utilize it
> >> in
> >>>>>> post-processing on the PSWS database to make available SID events,
> >>>>>> whistler and chorus events, and lightning location data, just to
> name
> >> a
> >>>>>> few.
> >>>>>>
> >>>>>> vtblank - an impulsive noise blanker that removes sferics and other
> >>>>>> impulsive noise from the VLF spectrum. It's useful in studying VLF
> >>>>>> transmitters and especially useful for VLF QSOs.
> >>>>>>
> >>>>>> vtam - used to extract the magnitude of a non-stationary signal
> like a
> >>>>>> sferic. You can examine the timing of sferics, whistlers, and the
> >> edges
> >>>> of
> >>>>>> timing signals like WWVB.
> >>>>>>
> >>>>>> vtpolar - generate a polar plot movie of a triple axis VLF stream.
> >> This
> >>>>>> allows you to see the bearing of incoming VLF signals like
> whistlers.
> >>>>>> Please watch the movie in the documentation!
> >>>>>>
> >>>>>> vtsid - automatically  detects sudden ionospheric disturbances with
> >> VLF
> >>>>>> transmitters like NAA. It generates a plot and a data file. The
> >>>> extraction
> >>>>>> utility can extract records from that data file.
> >>>>>>
> >>>>>> vtevent - automatically detects whistlers and chorus and generates a
> >>>>>> spectrogram plot, data file, and audio file of the event.
> >>>>>>
> >>>>>> vttoga - creates time of group arrival data from sferics the
> incoming
> >>>> VLF
> >>>>>> stream or a file.
> >>>>>>
> >>>>>> vtspot - used to generate lightning location data from vttoga data
> >> from
> >>>>>> VLF receivers around the world. The PSWS database can provide
> >> lightning
> >>>>>> location data using a network of receivers.
> >>>>>>
> >>>>>> vtping - used to detect meteor ping.
> >>>>>>
> >>>>>> vtwrite - writes a VLF stream, either from a ring buffer or network
> >>>>>> socket, to file for storage.
> >>>>>>
> >>>>>> vtread - reads a .vt file into a ring buffer or network socket.
> >>>>>>
> >>>>>> vtraw - extract raw audio froma VLF stream. This also removes the
> >>>>>> timestamping.
> >>>>>>
> >>>>>> vtwavex - extract a wav file from a VLF stream. This also removes
> the
> >>>>>> timestamping.
> >>>>>>
> >>>>>> These are just some examples of useful utilities of vlfrx-tools. A
> >>>> utility
> >>>>>> to convert a .vt file to DigitalRF/HDF5 can be written (however, I
> >>>> cannot
> >>>>>> support that). Also, vlfrx-tools does not generate IQ data normally.
> >>>> But,
> >>>>>> with such a toolset, so much can done. I would hope this convinces
> >> some
> >>>> of
> >>>>>> those who may be skeptical. In any case, I urge you to look at the
> >>>>>> documentation.
> >>>>>>
> >>>>>> Jonathan
> >>>>>> KC3EEY
> >>>>>>
> >>>>>> On Wed, 10 Mar 2021, Tom McDermott via TangerineSDR wrote:
> >>>>>>
> >>>>>>> Date: Wed, 10 Mar 2021 12:07:26 -0800
> >>>>>>> From: Tom McDermott via TangerineSDR <tangerinesdr at lists.tapr.org>
> >>>>>>> To: TAPR TangerineSDR Modular Software Defined Radio
> >>>>>>>     <tangerinesdr at lists.tapr.org>
> >>>>>>> Cc: Tom McDermott <tom.n5eg at gmail.com>
> >>>>>>> Subject: [TangerineSDR] VLF-RX tools
> >>>>>>>
> >>>>>>> Hi,  I don't have Jonathan Rizzo's direct email, so sending this to
> >> the
> >>>>>>> list.
> >>>>>>>
> >>>>>>> We talked a bit about the VLF receiver on the last conference call,
> >> and
> >>>>>> the
> >>>>>>> VLFRX tools.
> >>>>>>> I'd like to email with Jonathan directly.   It appears those tools
> >> have
> >>>>>> the
> >>>>>>> ability to
> >>>>>>> read from a file (and thus in Linux from a named pipe).  Gnuradio
> can
> >>>>>> write
> >>>>>>> to a named pipe
> >>>>>>> as well to DigitalRF (HDF5).   Perhaps this may provide a mechanism
> >> to
> >>>>>>> hook the receiver both to the Tangerine archiving method as well as
> >> to
> >>>>>>> VLFRX.?
> >>>>>>>
> >>>>>>> -- Tom, N5EG
> >>>>>>> n5eg at tapr.org
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> TangerineSDR mailing list
> >>>>>> TangerineSDR at lists.tapr.org
> >>>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> ----
> >>>>> Phil Erickson
> >>>>> phil.erickson at gmail.com
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> ----
> >>> Phil Erickson
> >>> phil.erickson at gmail.com
> >>>
> >>
> >
> >
> > --
> > ----
> > Phil Erickson
> > phil.erickson at gmail.com
> >
>


-- 
----
Phil Erickson
phil.erickson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20210311/dcfacfa2/attachment-0001.html>