[TangerineSDR] VLF-RX tools

[Jonathan]

Hi Phil,

When you mean "not always persistent across decades of measurements", do 
you mean sampling rate and sample length inconsistency across different 
data sources?

Yes, it is a purely real signal. There is a utility called vtresample that 
will resample the VLF stream and maintain timestamp alignment. And you are 
exactly right, there is a file in the source called vtlib.h which looks to 
define the timestamping format. Many of the utilities have this as an 
#include.

The author must not have felt a need to document the format. He's not 
affiliated with a university and I don't believe he's a licensed ham, 
however, he's an active VLF enthusiest. He must have written his tool set 
and used his format to maintain persistence in the data. I have always 
felt that his format was adequate, especially when using his stream 
manipulation utilities which align and maintain timestamping.

I wasn't sure about the reason for the insistance on using DigitalRF for 
VLF data analysis and storage on the TangerineSDR other than that's what 
the grants are paying for everyone to support. It's such a useful tool set 
that it seems like a waste not to utilize it.

Jonathan
KC3EEY

On Thu, 11 Mar 2021, Phil Erickson wrote:

> Date: Thu, 11 Mar 2021 19:12:58 -0500
> From: Phil Erickson <phil.erickson at gmail.com>
> To: Jonathan <emuman100 at gmail.com>
> Cc: TAPR TangerineSDR Modular Software Defined Radio
>     <tangerinesdr at lists.tapr.org>
> Subject: Re: [TangerineSDR] VLF-RX tools
> 
> Hi Jonathan,
>
>  We'll take a look.  For a purely real signal (e.g. cos(\omega t)), of
> course you would need exp(j \omega t) in I and exp(-j \omega t) in Q.  I
> believe DigitalRF may have the ability to capture purely real signals
> without the need to do 2X expansion into I and Q, but I'll ask.  Things
> like PCM audio streams are unfortunately not always persistent across
> decades of measurements (we know this sort of thing from hard experience at
> Haystack), so we choose HDF5 for long term persistence.
>
>  I don't have the time at the moment to wade through code, but if this is
> C code, it is likely there is a header file (".h") somewhere that contains
> the stream format.  You might post that here when it's found.  For this
> application, "derived from the source code" definitely needs to be replaced
> with a formal data specification and that could form the start of it.
>
> Cheers
> Phil W1PJE
>
> On Thu, Mar 11, 2021 at 7:02 PM Jonathan <emuman100 at gmail.com> wrote:
>
>> Hi Phil,
>>
>> The stream format is not documented, however, it can be derived from the
>> source code of many of the utility programs, like vtcard, vtcat, vtraw,
>> and vtwavex to name a few. It's single-channel or multi-channel pcm with
>> timestamping information. vtcat can combine single channel streams, say
>> from an E-Field receiver and two B-Field receivers into a multi-channel
>> stream and align timestamps of all channels.
>>
>> I'm not an expert at C by any means, and I haven't really looked at
>> understanding the stream format in the source code yet. For a conversion
>> utility, would the stream have to be copied and phase-shifted to generate
>> IQ data?
>>
>> vlfrx-tools captures and works with data in its original form. From the
>> soundcard, it's a pcm audio stream. There are utilities vtrtlsdr and
>> vtsdriq that read data from a RTL dongle or SDR-IQ receiver respectively.
>> vtsdriq can output either I and Q data in two channels or a single channel
>> stream.
>>
>> Jonathan
>> KC3EEY
>>
>> On Thu, 11 Mar 2021, Phil Erickson wrote:
>>
>>> Date: Thu, 11 Mar 2021 04:33:08 -0500
>>> From: Phil Erickson <phil.erickson at gmail.com>
>>> To: TAPR TangerineSDR Modular Software Defined Radio
>>>     <tangerinesdr at lists.tapr.org>
>>> Cc: Jonathan <emuman100 at gmail.com>
>>> Subject: Re: [TangerineSDR] VLF-RX tools
>>>
>>> Hi Jonathan,
>>>
>>>  I looked at the documentation (HTML page) but I cannot find a precise
>>> definition of the stream format.  Where is that located or could you
>>> summarize that?  It would then be possible for those of us familiar with
>>> DigitalRF, and in particular the Haystack team who maintains it, to
>> examine
>>> the ways in which a .vt file could be translated to DigitalRF or the
>> other
>>> way around.
>>>
>>> 73
>>> Phil W1PJE
>>>
>>> On Wed, Mar 10, 2021 at 10:46 PM Jonathan via TangerineSDR <
>>> tangerinesdr at lists.tapr.org> wrote:
>>>
>>>> Hi Tom,
>>>>
>>>> Thank you for looking at the documentation for vlfrx-tools! I'll list
>> some
>>>> useful utilities included in vlfrx-tools to try to illustrate how useful
>>>> they can be. A utility can be written to convert to DigitalRF/HDF5,
>>>> however, vlfrx-tools does not normally generate IQ data.
>>>>
>>>> For others on the list, the vlfrx-tools documentation is
>>>> available at http://abelian.org/vlfrx-tools/ and this also includes the
>>>> source code. Also, for the others on the list, I'd like to make the
>>>> argument that vlfrx-tools is advantageous because of its useful toolset
>> in
>>>> analysis and post-processing of both natural radio, amateur radio in the
>>>> VLF band, and VLF transmitters. Such tools don't exist (to my knowledge)
>>>> for formats like DigitalRF. Since vlfrx-tools is such a diverse and
>>>> usefull toolset that already exists, it's advantageous to utilize it in
>>>> post-processing on the PSWS database to make available SID events,
>>>> whistler and chorus events, and lightning location data, just to name a
>>>> few.
>>>>
>>>> vtblank - an impulsive noise blanker that removes sferics and other
>>>> impulsive noise from the VLF spectrum. It's useful in studying VLF
>>>> transmitters and especially useful for VLF QSOs.
>>>>
>>>> vtam - used to extract the magnitude of a non-stationary signal like a
>>>> sferic. You can examine the timing of sferics, whistlers, and the edges
>> of
>>>> timing signals like WWVB.
>>>>
>>>> vtpolar - generate a polar plot movie of a triple axis VLF stream. This
>>>> allows you to see the bearing of incoming VLF signals like whistlers.
>>>> Please watch the movie in the documentation!
>>>>
>>>> vtsid - automatically  detects sudden ionospheric disturbances with VLF
>>>> transmitters like NAA. It generates a plot and a data file. The
>> extraction
>>>> utility can extract records from that data file.
>>>>
>>>> vtevent - automatically detects whistlers and chorus and generates a
>>>> spectrogram plot, data file, and audio file of the event.
>>>>
>>>> vttoga - creates time of group arrival data from sferics the incoming
>> VLF
>>>> stream or a file.
>>>>
>>>> vtspot - used to generate lightning location data from vttoga data from
>>>> VLF receivers around the world. The PSWS database can provide lightning
>>>> location data using a network of receivers.
>>>>
>>>> vtping - used to detect meteor ping.
>>>>
>>>> vtwrite - writes a VLF stream, either from a ring buffer or network
>>>> socket, to file for storage.
>>>>
>>>> vtread - reads a .vt file into a ring buffer or network socket.
>>>>
>>>> vtraw - extract raw audio froma VLF stream. This also removes the
>>>> timestamping.
>>>>
>>>> vtwavex - extract a wav file from a VLF stream. This also removes the
>>>> timestamping.
>>>>
>>>> These are just some examples of useful utilities of vlfrx-tools. A
>> utility
>>>> to convert a .vt file to DigitalRF/HDF5 can be written (however, I
>> cannot
>>>> support that). Also, vlfrx-tools does not generate IQ data normally.
>> But,
>>>> with such a toolset, so much can done. I would hope this convinces some
>> of
>>>> those who may be skeptical. In any case, I urge you to look at the
>>>> documentation.
>>>>
>>>> Jonathan
>>>> KC3EEY
>>>>
>>>> On Wed, 10 Mar 2021, Tom McDermott via TangerineSDR wrote:
>>>>
>>>>> Date: Wed, 10 Mar 2021 12:07:26 -0800
>>>>> From: Tom McDermott via TangerineSDR <tangerinesdr at lists.tapr.org>
>>>>> To: TAPR TangerineSDR Modular Software Defined Radio
>>>>>     <tangerinesdr at lists.tapr.org>
>>>>> Cc: Tom McDermott <tom.n5eg at gmail.com>
>>>>> Subject: [TangerineSDR] VLF-RX tools
>>>>>
>>>>> Hi,  I don't have Jonathan Rizzo's direct email, so sending this to the
>>>>> list.
>>>>>
>>>>> We talked a bit about the VLF receiver on the last conference call, and
>>>> the
>>>>> VLFRX tools.
>>>>> I'd like to email with Jonathan directly.   It appears those tools have
>>>> the
>>>>> ability to
>>>>> read from a file (and thus in Linux from a named pipe).  Gnuradio can
>>>> write
>>>>> to a named pipe
>>>>> as well to DigitalRF (HDF5).   Perhaps this may provide a mechanism to
>>>>> hook the receiver both to the Tangerine archiving method as well as to
>>>>> VLFRX.?
>>>>>
>>>>> -- Tom, N5EG
>>>>> n5eg at tapr.org
>>>>>
>>>>
>>>> --
>>>> TangerineSDR mailing list
>>>> TangerineSDR at lists.tapr.org
>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>>
>>>
>>>
>>> --
>>> ----
>>> Phil Erickson
>>> phil.erickson at gmail.com
>>>
>>
>
>
> -- 
> ----
> Phil Erickson
> phil.erickson at gmail.com
>