[TangerineSDR] IGRF Python Package
Gerry Creager - NOAA Affiliate
gerry.creager at noaa.gov
Thu Apr 2 11:22:20 EDT 2020
Bill,
Howdy. Hat tip to Phil for pointing me in the right direction.
My experience and generally the software bits of choice for something like
this would be to see if we can implement using the RAMADDA and THREDDS
software from Unidata (UCAR). Generally, the data we're talking about is
amenable to being stored in HDF5 format, and that's a good fit for the
THREDDS catalogue system and RAMADDA indexing and selective, shopping-cart
access for researchers. THREDDS has an API that will support large
downloads, as well as aggregation and spatial segmentation.
Does this sound like a fit?
gerry n5jxs
On Thu, Apr 2, 2020 at 10:07 AM Phil Erickson <phil.erickson at gmail.com>
wrote:
> Hi Gerry,
>
> Bill is your single point of contact, as he's managing the data flow for
> the NSF DASI effort. I would synchronize with him and with Nathaniel and
> then the discussion can widen. FYI, Madrigal is a bit of a different beast
> having evolved over 4 decades from the ground based community - it is very
> instrument-centric rather than time and/or space coordinate-centric. It
> happens to have extensive use in the geospace community, but may or may not
> be the right choice here.
>
> Cheers
> Phil W1PJE
>
> On Thu, Apr 2, 2020 at 10:06 AM Gerry Creager - NOAA Affiliate <
> gerry.creager at noaa.gov> wrote:
>
>> Phil
>>
>> I've some experience deploying and maintaining data storage and
>> presentation. There are tools (although not Madrigal; I'll look) that I'm
>> already familiar with that support large datasets well. Multi-site
>> distribution is nearly trivial.
>>
>> What can I do to help?
>>
>> gerry n5jxs
>>
>> On Wed, Apr 1, 2020 at 1:20 PM Phil Erickson via TangerineSDR <
>> tangerinesdr at lists.tapr.org> wrote:
>>
>>> Hi Bill,
>>>
>>> There is something quick that comes to mind. There is a large
>>> difference between data provision through an interface (of whatever flavor)
>>> and data archiving as the long-term record. At MIT, Madrigal serves both
>>> functions for the ground based community, but that's merely one case. The
>>> project will most definitely need separate and sustained funding to keep
>>> the long term archive in place; it isn't going to happen by default, as you
>>> indicate, and Phase 1 is not going to have the resources to do that.
>>>
>>> As an example, in Madrigal's case, we are now about to deploy the
>>> following for the central long term archive:
>>>
>>> - server nodes in two physical locations, one of them a continuously
>>> manned industrial quality vault type location with zero downtime at the
>>> financial institution level;
>>> - each node itself is fully redundant with RAID 6 at a minimum and
>>> instant fail-over
>>> - each node exchanges updates every evening
>>> - network access on major back-haul pipes
>>>
>>> This of course is exactly what Google does for their Google File
>>> System or equivalent, except they tend to put their massive server farms
>>> very close to hydropower for cheap energy. As you said, there are many,
>>> many options out there. But the best one is one you control and maintain,
>>> with the associated cost of that maintenance - it's the only way to
>>> guarantee long term preservation. I simply do not trust any companies any
>>> longer in their promises. Nothing is really for free. (Look at the AWS
>>> price per byte now, and you pay for input AND output and storage. Gets
>>> non-economical in a hurry.)
>>>
>>> 73
>>> Phil
>>>
>>>
>>> On Wed, Apr 1, 2020 at 1:05 PM Engelke, Bill via TangerineSDR <
>>> tangerinesdr at lists.tapr.org> wrote:
>>>
>>>> This does raise several issues we will need to deal with at some
>>>> point. Some notes…
>>>>
>>>>
>>>>
>>>> - I have been pushing the software I am developing for the small
>>>> board computer (to go into the TangerineSDR) to my repository on Github
>>>> simply because it is the most straightforward way at this moment. It gives
>>>> me the ability to recover without major loss of work if the Odroid I am
>>>> using were to catastrophically fail; plus, if I were to drop dead next week
>>>> due to coronavirus or something (a realistic possibility, unfortunately),
>>>> somebody in the team may be able to make use of what I have done so far.
>>>> The software is heavily commented.
>>>> - It would indeed be a good idea for TAPR to have its own github
>>>> for this and related software to ensure its continued access to it.
>>>> - On a related note, we (here at UA) have extensively discussed how
>>>> to store data that will be collected from Personal Space WX Stations once
>>>> we have the network running. For Phase 1, we will simply put it on a UA
>>>> server and make it publicly available (and cross-referenced by the database
>>>> we are developing with NSF funding). It’s the longer term plan (Phase 2 and
>>>> beyond) that is tricky. Many options have been floated: Zenodo, Madrigal,
>>>> Grafana, Dropbox, OpenScienceDataCloud, Amazon Web Services, Microsoft
>>>> Azure, Box, etc. Each one has its pro and con, as well as a chorus of
>>>> supporters and detractors. A complicating factor on that is that we’d hope
>>>> to be able to enable some Big Data analysis approach; not to mention that
>>>> many of these solutions will get quite expensive at scale; which is OK *as
>>>> long as we have funding*… but this is part of the equation. As part
>>>> of the current project (albeit in year 2022), we plan to evaluate the
>>>> issues and make some recommendations (in what might become a Phase 2 NSF
>>>> proposal if all goes well). *We are most interested to hear
>>>> everyone’s thoughts on this *as we go forward; still, one thing we
>>>> don’t want to do is end up with a “horse-designed-by-a-committee” (i.e., a
>>>> camel) result. (Also, based on past experience, some neat new solution
>>>> could become available in the next 2 to 3 years that would be even better
>>>> than current options.)
>>>>
>>>>
>>>>
>>>> -73- Bill AB4EJ
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* TangerineSDR <tangerinesdr-bounces at lists.tapr.org> *On Behalf
>>>> Of *Phil Erickson via TangerineSDR
>>>> *Sent:* Wednesday, April 1, 2020 10:50 AM
>>>> *To:* TAPR TangerineSDR Modular Software Defined Radio <
>>>> tangerinesdr at lists.tapr.org>
>>>> *Cc:* Phil Erickson <phil.erickson at gmail.com>; Stephen Roland Kaeppler
>>>> <skaeppl at clemson.edu>
>>>> *Subject:* Re: [TangerineSDR] IGRF Python Package
>>>>
>>>>
>>>>
>>>> Hi all,
>>>>
>>>>
>>>>
>>>> I strongly recommend that this project
>>>>
>>>>
>>>>
>>>> a) use revision control
>>>>
>>>> b) host its own SVN / git / etc. server
>>>>
>>>>
>>>>
>>>> Relying on things like GitHub is not safe in the long term, because
>>>> corporate decisions can change abruptly on what is offered free to the
>>>> community and what is not. Also, many of the "free" services make
>>>> absolutely no guarantees about where any of the data is backed up, what the
>>>> backup schedule is, etc.
>>>>
>>>>
>>>>
>>>> I relate these pieces of advice from painful experience at MIT
>>>> Haystack. Our group believed Dropbox when they rolled out a storage plan
>>>> with very large upper bound limits for MIT. A couple years ago, they
>>>> abruptly cancelled that policy and assigned a very large per year fee for
>>>> the storage we had been using. I'm still trying to figure out how to
>>>> extract 100s of TB of data from them, and some of it may be lost for good.
>>>>
>>>>
>>>>
>>>> 73
>>>>
>>>> Phil W1PJE
>>>>
>>>>
>>>>
>>>> On Wed, Apr 1, 2020 at 11:45 AM Tom McDermott via TangerineSDR <
>>>> tangerinesdr at lists.tapr.org> wrote:
>>>>
>>>> Hi Steve - thanks for the offer !
>>>>
>>>>
>>>>
>>>> This brings up a general question: should TangerineSDR / PSWS host
>>>> this kind of software someplace?
>>>>
>>>> One could imagine that over time there may be an increasing number of
>>>> software utilities and packages of
>>>>
>>>> interest to the group, it would be nice to have them accessible through
>>>> one link or portal (ideally with revision control ! ).
>>>>
>>>>
>>>>
>>>> Is this something the group should consider? If so, is github the
>>>> right solution?, a page of links?, something else?
>>>>
>>>>
>>>>
>>>> -- Tom, N5EG
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Apr 1, 2020 at 5:57 AM Stephen Roland Kaeppler via TangerineSDR
>>>> <tangerinesdr at lists.tapr.org> wrote:
>>>>
>>>> Hi All-
>>>>
>>>> I will add a comment if it is of interest, that I wrote some code a
>>>> while back in python that directly calculates the magnetic field using the
>>>> IGRF coefficients and derivatives (useful in a ray tracer). If you guys
>>>> want that, I could package it up and send it over to anyone interested.
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>> ------------------------------------------------------------
>>>> Stephen R. Kaeppler, Ph.D.
>>>> Assistant Professor
>>>> Department of Physics and Astronomy
>>>> Clemson University
>>>> Clemson, SC 29634
>>>> Email: skaeppl at clemson.edu
>>>> Phone: 864-656-4275
>>>> Web: http://science.clemson.edu/kaeppler/
>>>> Amateur Radio Callsign: AD0AE
>>>> ------------------------------------------------------------
>>>>
>>>>
>>>> --
>>>> TangerineSDR mailing list
>>>> TangerineSDR at lists.tapr.org
>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>>
>>>> --
>>>> TangerineSDR mailing list
>>>> TangerineSDR at lists.tapr.org
>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ----
>>>> Phil Erickson
>>>> phil.erickson at gmail.com
>>>> --
>>>> TangerineSDR mailing list
>>>> TangerineSDR at lists.tapr.org
>>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>>
>>>
>>>
>>> --
>>> ----
>>> Phil Erickson
>>> phil.erickson at gmail.com
>>> --
>>> TangerineSDR mailing list
>>> TangerineSDR at lists.tapr.org
>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>
>>
>>
>> --
>> Gerry Creager
>> NSSL/CIMMS
>> 405.325.6371
>> ++++++++++++++++++++++
>> *The way to get started is to quit talking and begin doing.*
>> * Walt Disney*
>>
>
>
> --
> ----
> Phil Erickson
> phil.erickson at gmail.com
>
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
*The way to get started is to quit talking and begin doing.*
* Walt Disney*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20200402/66630ba0/attachment-0001.html>
More information about the TangerineSDR
mailing list