[TangerineSDR] VLF-RX tools

Jonathan emuman100 at gmail.com
Thu Mar 11 20:43:59 EST 2021 

Hi Phil,

I didn't see the message from Ryan for some reason, but I'm very happy to 
hear this. Thank you for having this looked into!

Jonathan
KC3EEY

On Thu, 11 Mar 2021, Phil Erickson wrote:

> Date: Thu, 11 Mar 2021 20:36:06 -0500
> From: Phil Erickson <phil.erickson at gmail.com>
> To: Jonathan <emuman100 at gmail.com>
> Cc: TAPR TangerineSDR Modular Software Defined Radio
>     <tangerinesdr at lists.tapr.org>
> Subject: Re: [TangerineSDR] VLF-RX tools
> 
> Hi Jonathan,
>
>  As Ryan said, some kind of a tool as he described it which reads from a
> pipe should be straightforward.  The example you've given is a way to get
> started, and I'll talk with him later about getting that implemented.  It's
> a use case we want to support.
>
> Cheers
> Phil W1PJE
>
> On Thu, Mar 11, 2021 at 8:16 PM Jonathan <emuman100 at gmail.com> wrote:
>
>> Hi Phil,
>>
>> I understand now. The same issue has been encountered in seismology with
>> the many different formats to store seismic wave data. The current winners
>> now seem to be SAQ and miniseed formats, where miniseed has the ability to
>> store metadata, such as instrument type and calibration data required for
>> that seismic instrument. Both formats and others have detailed spec
>> documentation. (I believe you can store metadata in the SAQ format as
>> well, but I'm not sure)
>>
>> In that case, since purely real streams can be supported, is there a
>> possibility of a conversion tool becoming a reality? I cannot support
>> writing a conversion tool, but others seem to want such a utility. If you
>> go to the root of the vlfrx-tools site and navigate to "live streams",
>> then on any one of the streams, click "events", a page will come up that
>> shows whistler and chorus events (with a lot of false triggers due to
>> motors starting up). With each event, you can download the .vt file and
>> use it as an example file for a conversion utility.
>>
>> Jonathan
>> KC3EEY
>>
>>
>> On Thu, 11 Mar 2021, Phil Erickson wrote:
>>
>>> Date: Thu, 11 Mar 2021 19:57:54 -0500
>>> From: Phil Erickson <phil.erickson at gmail.com>
>>> To: Jonathan <emuman100 at gmail.com>
>>> Cc: TAPR TangerineSDR Modular Software Defined Radio
>>>     <tangerinesdr at lists.tapr.org>
>>> Subject: Re: [TangerineSDR] VLF-RX tools
>>>
>>> Hi Jonathan,
>>>
>>>  There are only a very few formats that are designed for long term and
>>> cross-platform "survival" across many different operating systems and
>>> languages, and which have enough embedded metadata to describe the data
>> in
>>> enough detail to allow applications that were not thought of at the time
>> of
>>> recording.  HDF5 is the product of a huge amount of work - decades of
>>> person-years - to enable that sort of survivability.  We therefore use it
>>> as the carrier, with DigitalRF as the particular data structure
>> implemented
>>> in it.
>>>
>>>  I think Ryan's separate message is a workflow that would allow all the
>>> good toolsets to be implemented without changing the way that the
>> recording
>>> happens.  Note as well that we do support purely real streams (no need to
>>> always have I and Q) and getting a good use case which implements that is
>>> on our list of ways we want to move forward with the format.
>>>
>>> Cheers
>>> Phil W1PJE
>>>
>>> On Thu, Mar 11, 2021 at 7:47 PM Jonathan <emuman100 at gmail.com> wrote:
>>>
>>>> Hi Phil,
>>>>
>>>> When you mean "not always persistent across decades of measurements", do
>>>> you mean sampling rate and sample length inconsistency across different
>>>> data sources?
>>>>
>>>> Yes, it is a purely real signal. There is a utility called vtresample
>> that
>>>> will resample the VLF stream and maintain timestamp alignment. And you
>> are
>>>> exactly right, there is a file in the source called vtlib.h which looks
>> to
>>>> define the timestamping format. Many of the utilities have this as an
>>>> #include.
>>>>
>>>> The author must not have felt a need to document the format. He's not
>>>> affiliated with a university and I don't believe he's a licensed ham,
>>>> however, he's an active VLF enthusiest. He must have written his tool
>> set
>>>> and used his format to maintain persistence in the data. I have always
>>>> felt that his format was adequate, especially when using his stream
>>>> manipulation utilities which align and maintain timestamping.
>>>>
>>>> I wasn't sure about the reason for the insistance on using DigitalRF for
>>>> VLF data analysis and storage on the TangerineSDR other than that's what
>>>> the grants are paying for everyone to support. It's such a useful tool
>> set
>>>> that it seems like a waste not to utilize it.
>>>>
>>>> Jonathan
>>>> KC3EEY
>>>>
>>>> On Thu, 11 Mar 2021, Phil Erickson wrote:
>>>>
>>>>> Date: Thu, 11 Mar 2021 19:12:58 -0500
>>>>> From: Phil Erickson <phil.erickson at gmail.com>
>>>>> To: Jonathan <emuman100 at gmail.com>
>>>>> Cc: TAPR TangerineSDR Modular Software Defined Radio
>>>>>     <tangerinesdr at lists.tapr.org>
>>>>> Subject: Re: [TangerineSDR] VLF-RX tools
>>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>>  We'll take a look.  For a purely real signal (e.g. cos(\omega t)), of
>>>>> course you would need exp(j \omega t) in I and exp(-j \omega t) in Q.
>> I
>>>>> believe DigitalRF may have the ability to capture purely real signals
>>>>> without the need to do 2X expansion into I and Q, but I'll ask.  Things
>>>>> like PCM audio streams are unfortunately not always persistent across
>>>>> decades of measurements (we know this sort of thing from hard
>> experience
>>>> at
>>>>> Haystack), so we choose HDF5 for long term persistence.
>>>>>
>>>>>  I don't have the time at the moment to wade through code, but if this
>> is
>>>>> C code, it is likely there is a header file (".h") somewhere that
>>>> contains
>>>>> the stream format.  You might post that here when it's found.  For this
>>>>> application, "derived from the source code" definitely needs to be
>>>> replaced
>>>>> with a formal data specification and that could form the start of it.
>>>>>
>>>>> Cheers
>>>>> Phil W1PJE
>>>>>
>>>>> On Thu, Mar 11, 2021 at 7:02 PM Jonathan <emuman100 at gmail.com> wrote:
>>>>>
>>>>>> Hi Phil,
>>>>>>
>>>>>> The stream format is not documented, however, it can be derived from
>> the
>>>>>> source code of many of the utility programs, like vtcard, vtcat,
>> vtraw,
>>>>>> and vtwavex to name a few. It's single-channel or multi-channel pcm
>> with
>>>>>> timestamping information. vtcat can combine single channel streams,
>> say
>>>>>> from an E-Field receiver and two B-Field receivers into a
>> multi-channel
>>>>>> stream and align timestamps of all channels.
>>>>>>
>>>>>> I'm not an expert at C by any means, and I haven't really looked at
>>>>>> understanding the stream format in the source code yet. For a
>> conversion
>>>>>> utility, would the stream have to be copied and phase-shifted to
>>>> generate
>>>>>> IQ data?
>>>>>>
>>>>>> vlfrx-tools captures and works with data in its original form. From
>> the
>>>>>> soundcard, it's a pcm audio stream. There are utilities vtrtlsdr and
>>>>>> vtsdriq that read data from a RTL dongle or SDR-IQ receiver
>>>> respectively.
>>>>>> vtsdriq can output either I and Q data in two channels or a single
>>>> channel
>>>>>> stream.
>>>>>>
>>>>>> Jonathan
>>>>>> KC3EEY
>>>>>>
>>>>>> On Thu, 11 Mar 2021, Phil Erickson wrote:
>>>>>>
>>>>>>> Date: Thu, 11 Mar 2021 04:33:08 -0500
>>>>>>> From: Phil Erickson <phil.erickson at gmail.com>
>>>>>>> To: TAPR TangerineSDR Modular Software Defined Radio
>>>>>>>     <tangerinesdr at lists.tapr.org>
>>>>>>> Cc: Jonathan <emuman100 at gmail.com>
>>>>>>> Subject: Re: [TangerineSDR] VLF-RX tools
>>>>>>>
>>>>>>> Hi Jonathan,
>>>>>>>
>>>>>>>  I looked at the documentation (HTML page) but I cannot find a
>> precise
>>>>>>> definition of the stream format.  Where is that located or could you
>>>>>>> summarize that?  It would then be possible for those of us familiar
>>>> with
>>>>>>> DigitalRF, and in particular the Haystack team who maintains it, to
>>>>>> examine
>>>>>>> the ways in which a .vt file could be translated to DigitalRF or the
>>>>>> other
>>>>>>> way around.
>>>>>>>
>>>>>>> 73
>>>>>>> Phil W1PJE
>>>>>>>
>>>>>>> On Wed, Mar 10, 2021 at 10:46 PM Jonathan via TangerineSDR <
>>>>>>> tangerinesdr at lists.tapr.org> wrote:
>>>>>>>
>>>>>>>> Hi Tom,
>>>>>>>>
>>>>>>>> Thank you for looking at the documentation for vlfrx-tools! I'll
>> list
>>>>>> some
>>>>>>>> useful utilities included in vlfrx-tools to try to illustrate how
>>>> useful
>>>>>>>> they can be. A utility can be written to convert to DigitalRF/HDF5,
>>>>>>>> however, vlfrx-tools does not normally generate IQ data.
>>>>>>>>
>>>>>>>> For others on the list, the vlfrx-tools documentation is
>>>>>>>> available at http://abelian.org/vlfrx-tools/ and this also includes
>>>> the
>>>>>>>> source code. Also, for the others on the list, I'd like to make the
>>>>>>>> argument that vlfrx-tools is advantageous because of its useful
>>>> toolset
>>>>>> in
>>>>>>>> analysis and post-processing of both natural radio, amateur radio in
>>>> the
>>>>>>>> VLF band, and VLF transmitters. Such tools don't exist (to my
>>>> knowledge)
>>>>>>>> for formats like DigitalRF. Since vlfrx-tools is such a diverse and
>>>>>>>> usefull toolset that already exists, it's advantageous to utilize it
>>>> in
>>>>>>>> post-processing on the PSWS database to make available SID events,
>>>>>>>> whistler and chorus events, and lightning location data, just to
>> name
>>>> a
>>>>>>>> few.
>>>>>>>>
>>>>>>>> vtblank - an impulsive noise blanker that removes sferics and other
>>>>>>>> impulsive noise from the VLF spectrum. It's useful in studying VLF
>>>>>>>> transmitters and especially useful for VLF QSOs.
>>>>>>>>
>>>>>>>> vtam - used to extract the magnitude of a non-stationary signal
>> like a
>>>>>>>> sferic. You can examine the timing of sferics, whistlers, and the
>>>> edges
>>>>>> of
>>>>>>>> timing signals like WWVB.
>>>>>>>>
>>>>>>>> vtpolar - generate a polar plot movie of a triple axis VLF stream.
>>>> This
>>>>>>>> allows you to see the bearing of incoming VLF signals like
>> whistlers.
>>>>>>>> Please watch the movie in the documentation!
>>>>>>>>
>>>>>>>> vtsid - automatically  detects sudden ionospheric disturbances with
>>>> VLF
>>>>>>>> transmitters like NAA. It generates a plot and a data file. The
>>>>>> extraction
>>>>>>>> utility can extract records from that data file.
>>>>>>>>
>>>>>>>> vtevent - automatically detects whistlers and chorus and generates a
>>>>>>>> spectrogram plot, data file, and audio file of the event.
>>>>>>>>
>>>>>>>> vttoga - creates time of group arrival data from sferics the
>> incoming
>>>>>> VLF
>>>>>>>> stream or a file.
>>>>>>>>
>>>>>>>> vtspot - used to generate lightning location data from vttoga data
>>>> from
>>>>>>>> VLF receivers around the world. The PSWS database can provide
>>>> lightning
>>>>>>>> location data using a network of receivers.
>>>>>>>>
>>>>>>>> vtping - used to detect meteor ping.
>>>>>>>>
>>>>>>>> vtwrite - writes a VLF stream, either from a ring buffer or network
>>>>>>>> socket, to file for storage.
>>>>>>>>
>>>>>>>> vtread - reads a .vt file into a ring buffer or network socket.
>>>>>>>>
>>>>>>>> vtraw - extract raw audio froma VLF stream. This also removes the
>>>>>>>> timestamping.
>>>>>>>>
>>>>>>>> vtwavex - extract a wav file from a VLF stream. This also removes
>> the
>>>>>>>> timestamping.
>>>>>>>>
>>>>>>>> These are just some examples of useful utilities of vlfrx-tools. A
>>>>>> utility
>>>>>>>> to convert a .vt file to DigitalRF/HDF5 can be written (however, I
>>>>>> cannot
>>>>>>>> support that). Also, vlfrx-tools does not generate IQ data normally.
>>>>>> But,
>>>>>>>> with such a toolset, so much can done. I would hope this convinces
>>>> some
>>>>>> of
>>>>>>>> those who may be skeptical. In any case, I urge you to look at the
>>>>>>>> documentation.
>>>>>>>>
>>>>>>>> Jonathan
>>>>>>>> KC3EEY
>>>>>>>>
>>>>>>>> On Wed, 10 Mar 2021, Tom McDermott via TangerineSDR wrote:
>>>>>>>>
>>>>>>>>> Date: Wed, 10 Mar 2021 12:07:26 -0800
>>>>>>>>> From: Tom McDermott via TangerineSDR <tangerinesdr at lists.tapr.org>
>>>>>>>>> To: TAPR TangerineSDR Modular Software Defined Radio
>>>>>>>>>     <tangerinesdr at lists.tapr.org>
>>>>>>>>> Cc: Tom McDermott <tom.n5eg at gmail.com>
>>>>>>>>> Subject: [TangerineSDR] VLF-RX tools
>>>>>>>>>
>>>>>>>>> Hi,  I don't have Jonathan Rizzo's direct email, so sending this to
>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> We talked a bit about the VLF receiver on the last conference call,
>>>> and
>>>>>>>> the
>>>>>>>>> VLFRX tools.
>>>>>>>>> I'd like to email with Jonathan directly.   It appears those tools
>>>> have
>>>>>>>> the
>>>>>>>>> ability to
>>>>>>>>> read from a file (and thus in Linux from a named pipe).  Gnuradio
>> can
>>>>>>>> write
>>>>>>>>> to a named pipe
>>>>>>>>> as well to DigitalRF (HDF5).   Perhaps this may provide a mechanism
>>>> to
>>>>>>>>> hook the receiver both to the Tangerine archiving method as well as
>>>> to
>>>>>>>>> VLFRX.?
>>>>>>>>>
>>>>>>>>> -- Tom, N5EG
>>>>>>>>> n5eg at tapr.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> TangerineSDR mailing list
>>>>>>>> TangerineSDR at lists.tapr.org
>>>>>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ----
>>>>>>> Phil Erickson
>>>>>>> phil.erickson at gmail.com
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ----
>>>>> Phil Erickson
>>>>> phil.erickson at gmail.com
>>>>>
>>>>
>>>
>>>
>>> --
>>> ----
>>> Phil Erickson
>>> phil.erickson at gmail.com
>>>
>>
>
>
> -- 
> ----
> Phil Erickson
> phil.erickson at gmail.com
>

More information about the TangerineSDR mailing list