[TangerineSDR] IGRF Python Package
Phil Erickson
phil.erickson at gmail.com
Thu Apr 2 11:07:23 EDT 2020
Hi Gerry,
Bill is your single point of contact, as he's managing the data flow for
the NSF DASI effort. I would synchronize with him and with Nathaniel and
then the discussion can widen. FYI, Madrigal is a bit of a different beast
having evolved over 4 decades from the ground based community - it is very
instrument-centric rather than time and/or space coordinate-centric. It
happens to have extensive use in the geospace community, but may or may not
be the right choice here.
Cheers
Phil W1PJE
On Thu, Apr 2, 2020 at 10:06 AM Gerry Creager - NOAA Affiliate <
gerry.creager at noaa.gov> wrote:
> Phil
>
> I've some experience deploying and maintaining data storage and
> presentation. There are tools (although not Madrigal; I'll look) that I'm
> already familiar with that support large datasets well. Multi-site
> distribution is nearly trivial.
>
> What can I do to help?
>
> gerry n5jxs
>
> On Wed, Apr 1, 2020 at 1:20 PM Phil Erickson via TangerineSDR <
> tangerinesdr at lists.tapr.org> wrote:
>
>> Hi Bill,
>>
>> There is something quick that comes to mind. There is a large
>> difference between data provision through an interface (of whatever flavor)
>> and data archiving as the long-term record. At MIT, Madrigal serves both
>> functions for the ground based community, but that's merely one case. The
>> project will most definitely need separate and sustained funding to keep
>> the long term archive in place; it isn't going to happen by default, as you
>> indicate, and Phase 1 is not going to have the resources to do that.
>>
>> As an example, in Madrigal's case, we are now about to deploy the
>> following for the central long term archive:
>>
>> - server nodes in two physical locations, one of them a continuously
>> manned industrial quality vault type location with zero downtime at the
>> financial institution level;
>> - each node itself is fully redundant with RAID 6 at a minimum and
>> instant fail-over
>> - each node exchanges updates every evening
>> - network access on major back-haul pipes
>>
>> This of course is exactly what Google does for their Google File System
>> or equivalent, except they tend to put their massive server farms very
>> close to hydropower for cheap energy. As you said, there are many, many
>> options out there. But the best one is one you control and maintain, with
>> the associated cost of that maintenance - it's the only way to guarantee
>> long term preservation. I simply do not trust any companies any longer in
>> their promises. Nothing is really for free. (Look at the AWS price per
>> byte now, and you pay for input AND output and storage. Gets
>> non-economical in a hurry.)
>>
>> 73
>> Phil
>>
>>
>> On Wed, Apr 1, 2020 at 1:05 PM Engelke, Bill via TangerineSDR <
>> tangerinesdr at lists.tapr.org> wrote:
>>
>>> This does raise several issues we will need to deal with at some point.
>>> Some notes…
>>>
>>>
>>>
>>> - I have been pushing the software I am developing for the small
>>> board computer (to go into the TangerineSDR) to my repository on Github
>>> simply because it is the most straightforward way at this moment. It gives
>>> me the ability to recover without major loss of work if the Odroid I am
>>> using were to catastrophically fail; plus, if I were to drop dead next week
>>> due to coronavirus or something (a realistic possibility, unfortunately),
>>> somebody in the team may be able to make use of what I have done so far.
>>> The software is heavily commented.
>>> - It would indeed be a good idea for TAPR to have its own github for
>>> this and related software to ensure its continued access to it.
>>> - On a related note, we (here at UA) have extensively discussed how
>>> to store data that will be collected from Personal Space WX Stations once
>>> we have the network running. For Phase 1, we will simply put it on a UA
>>> server and make it publicly available (and cross-referenced by the database
>>> we are developing with NSF funding). It’s the longer term plan (Phase 2 and
>>> beyond) that is tricky. Many options have been floated: Zenodo, Madrigal,
>>> Grafana, Dropbox, OpenScienceDataCloud, Amazon Web Services, Microsoft
>>> Azure, Box, etc. Each one has its pro and con, as well as a chorus of
>>> supporters and detractors. A complicating factor on that is that we’d hope
>>> to be able to enable some Big Data analysis approach; not to mention that
>>> many of these solutions will get quite expensive at scale; which is OK *as
>>> long as we have funding*… but this is part of the equation. As part
>>> of the current project (albeit in year 2022), we plan to evaluate the
>>> issues and make some recommendations (in what might become a Phase 2 NSF
>>> proposal if all goes well). *We are most interested to hear
>>> everyone’s thoughts on this *as we go forward; still, one thing we
>>> don’t want to do is end up with a “horse-designed-by-a-committee” (i.e., a
>>> camel) result. (Also, based on past experience, some neat new solution
>>> could become available in the next 2 to 3 years that would be even better
>>> than current options.)
>>>
>>>
>>>
>>> -73- Bill AB4EJ
>>>
>>>
>>>
>>>
>>>
>>> *From:* TangerineSDR <tangerinesdr-bounces at lists.tapr.org> *On Behalf
>>> Of *Phil Erickson via TangerineSDR
>>> *Sent:* Wednesday, April 1, 2020 10:50 AM
>>> *To:* TAPR TangerineSDR Modular Software Defined Radio <
>>> tangerinesdr at lists.tapr.org>
>>> *Cc:* Phil Erickson <phil.erickson at gmail.com>; Stephen Roland Kaeppler <
>>> skaeppl at clemson.edu>
>>> *Subject:* Re: [TangerineSDR] IGRF Python Package
>>>
>>>
>>>
>>> Hi all,
>>>
>>>
>>>
>>> I strongly recommend that this project
>>>
>>>
>>>
>>> a) use revision control
>>>
>>> b) host its own SVN / git / etc. server
>>>
>>>
>>>
>>> Relying on things like GitHub is not safe in the long term, because
>>> corporate decisions can change abruptly on what is offered free to the
>>> community and what is not. Also, many of the "free" services make
>>> absolutely no guarantees about where any of the data is backed up, what the
>>> backup schedule is, etc.
>>>
>>>
>>>
>>> I relate these pieces of advice from painful experience at MIT
>>> Haystack. Our group believed Dropbox when they rolled out a storage plan
>>> with very large upper bound limits for MIT. A couple years ago, they
>>> abruptly cancelled that policy and assigned a very large per year fee for
>>> the storage we had been using. I'm still trying to figure out how to
>>> extract 100s of TB of data from them, and some of it may be lost for good.
>>>
>>>
>>>
>>> 73
>>>
>>> Phil W1PJE
>>>
>>>
>>>
>>> On Wed, Apr 1, 2020 at 11:45 AM Tom McDermott via TangerineSDR <
>>> tangerinesdr at lists.tapr.org> wrote:
>>>
>>> Hi Steve - thanks for the offer !
>>>
>>>
>>>
>>> This brings up a general question: should TangerineSDR / PSWS host this
>>> kind of software someplace?
>>>
>>> One could imagine that over time there may be an increasing number of
>>> software utilities and packages of
>>>
>>> interest to the group, it would be nice to have them accessible through
>>> one link or portal (ideally with revision control ! ).
>>>
>>>
>>>
>>> Is this something the group should consider? If so, is github the right
>>> solution?, a page of links?, something else?
>>>
>>>
>>>
>>> -- Tom, N5EG
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Apr 1, 2020 at 5:57 AM Stephen Roland Kaeppler via TangerineSDR <
>>> tangerinesdr at lists.tapr.org> wrote:
>>>
>>> Hi All-
>>>
>>> I will add a comment if it is of interest, that I wrote some code a
>>> while back in python that directly calculates the magnetic field using the
>>> IGRF coefficients and derivatives (useful in a ray tracer). If you guys
>>> want that, I could package it up and send it over to anyone interested.
>>>
>>> Thanks,
>>> Steve
>>>
>>> ------------------------------------------------------------
>>> Stephen R. Kaeppler, Ph.D.
>>> Assistant Professor
>>> Department of Physics and Astronomy
>>> Clemson University
>>> Clemson, SC 29634
>>> Email: skaeppl at clemson.edu
>>> Phone: 864-656-4275
>>> Web: http://science.clemson.edu/kaeppler/
>>> Amateur Radio Callsign: AD0AE
>>> ------------------------------------------------------------
>>>
>>>
>>> --
>>> TangerineSDR mailing list
>>> TangerineSDR at lists.tapr.org
>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>
>>> --
>>> TangerineSDR mailing list
>>> TangerineSDR at lists.tapr.org
>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>
>>>
>>>
>>>
>>> --
>>>
>>> ----
>>> Phil Erickson
>>> phil.erickson at gmail.com
>>> --
>>> TangerineSDR mailing list
>>> TangerineSDR at lists.tapr.org
>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>
>>
>>
>> --
>> ----
>> Phil Erickson
>> phil.erickson at gmail.com
>> --
>> TangerineSDR mailing list
>> TangerineSDR at lists.tapr.org
>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>
>
>
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> *The way to get started is to quit talking and begin doing.*
> * Walt Disney*
>
--
----
Phil Erickson
phil.erickson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20200402/d330eb62/attachment-0001.html>
More information about the TangerineSDR
mailing list