[TangerineSDR] IGRF Python Package

Thu Apr 2 10:05:49 EDT 2020

Phil

I've some experience deploying and maintaining data storage and
presentation. There are tools (although not Madrigal; I'll look) that I'm
already familiar with that support large datasets well. Multi-site
distribution is nearly trivial.

What can I do to help?

gerry n5jxs

On Wed, Apr 1, 2020 at 1:20 PM Phil Erickson via TangerineSDR <
tangerinesdr at lists.tapr.org> wrote:

> Hi Bill,
>
>   There is something quick that comes to mind.  There is a large
> difference between data provision through an interface (of whatever flavor)
> and data archiving as the long-term record.  At MIT, Madrigal serves both
> functions for the ground based community, but that's merely one case.  The
> project will most definitely need separate and sustained funding to keep
> the long term archive in place; it isn't going to happen by default, as you
> indicate, and Phase 1 is not going to have the resources to do that.
>
>   As an example, in Madrigal's case, we are now about to deploy the
> following for the central long term archive:
>
> - server nodes in two physical locations, one of them a continuously
> manned industrial quality vault type location with zero downtime at the
> financial institution level;
> - each node itself is fully redundant with RAID 6 at a minimum and instant
> fail-over
> - each node exchanges updates every evening
> - network access on major back-haul pipes
>
>   This of course is exactly what Google does for their Google File System
> or equivalent, except they tend to put their massive server farms very
> close to hydropower for cheap energy.  As you said, there are many, many
> options out there.  But the best one is one you control and maintain, with
> the associated cost of that maintenance - it's the only way to guarantee
> long term preservation.  I simply do not trust any companies any longer in
> their promises.  Nothing is really for free.  (Look at the AWS price per
> byte now, and you pay for input AND output and storage.  Gets
> non-economical in a hurry.)
>
> 73
> Phil
>
>
> On Wed, Apr 1, 2020 at 1:05 PM Engelke, Bill via TangerineSDR <
> tangerinesdr at lists.tapr.org> wrote:
>
>> This does raise several issues we will need to deal with at some point.
>> Some notes…
>>
>>
>>
>>    - I have been pushing the software I am developing for the small
>>    board computer (to go into the TangerineSDR) to my repository on Github
>>    simply because it is the most straightforward way at this moment. It gives
>>    me the ability to recover without major loss of work if the Odroid I am
>>    using were to catastrophically fail; plus, if I were to drop dead next week
>>    due to coronavirus or something (a realistic possibility, unfortunately),
>>    somebody in the team may be able to make use of what I have done so far.
>>    The software is heavily commented.
>>    - It would indeed be a good idea for TAPR to have its own github for
>>    this and related software to ensure its continued  access to it.
>>    - On a related note, we (here at UA) have extensively discussed how
>>    to store data that will be collected from Personal Space WX Stations once
>>    we have the network running.  For Phase 1, we will simply put it on a UA
>>    server and make it publicly available (and cross-referenced by the database
>>    we are developing with NSF funding). It’s the longer term plan (Phase 2 and
>>    beyond) that is tricky. Many options have been floated: Zenodo, Madrigal,
>>    Grafana, Dropbox, OpenScienceDataCloud, Amazon Web Services, Microsoft
>>    Azure, Box, etc. Each one has its pro and con, as well as a chorus of
>>    supporters and detractors. A complicating factor on that is that we’d hope
>>    to be able to enable some Big Data analysis approach; not to mention that
>>    many of these solutions will get quite expensive at scale; which is OK *as
>>    long as we have funding*… but this is part of the equation. As part
>>    of the current project (albeit in year 2022), we plan to evaluate the
>>    issues and make some recommendations (in what might become a Phase 2 NSF
>>    proposal if all goes well). *We are most interested to hear
>>    everyone’s thoughts on this *as we go forward; still, one thing we
>>    don’t want to do is end up with a “horse-designed-by-a-committee” (i.e., a
>>    camel) result. (Also, based on past experience, some neat new solution
>>    could become available in the next 2 to 3 years that would be even better
>>    than current options.)
>>
>>
>>
>> -73- Bill AB4EJ
>>
>>
>>
>>
>>
>> *From:* TangerineSDR <tangerinesdr-bounces at lists.tapr.org> *On Behalf Of
>> *Phil Erickson via TangerineSDR
>> *Sent:* Wednesday, April 1, 2020 10:50 AM
>> *To:* TAPR TangerineSDR Modular Software Defined Radio <
>> tangerinesdr at lists.tapr.org>
>> *Cc:* Phil Erickson <phil.erickson at gmail.com>; Stephen Roland Kaeppler <
>> skaeppl at clemson.edu>
>> *Subject:* Re: [TangerineSDR] IGRF Python Package
>>
>>
>>
>> Hi all,
>>
>>
>>
>>   I strongly recommend that this project
>>
>>
>>
>> a) use revision control
>>
>> b) host its own SVN / git / etc. server
>>
>>
>>
>>   Relying on things like GitHub is not safe in the long term, because
>> corporate decisions can change abruptly on what is offered free to the
>> community and what is not.  Also, many of the "free" services make
>> absolutely no guarantees about where any of the data is backed up, what the
>> backup schedule is, etc.
>>
>>
>>
>>   I relate these pieces of advice from painful experience at MIT
>> Haystack.  Our group believed Dropbox when they rolled out a storage plan
>> with very large upper bound limits for MIT.  A couple years ago, they
>> abruptly cancelled that policy and assigned a very large per year fee for
>> the storage we had been using.  I'm still trying to figure out how to
>> extract 100s of TB of data from them, and some of it may be lost for good.
>>
>>
>>
>> 73
>>
>> Phil W1PJE
>>
>>
>>
>> On Wed, Apr 1, 2020 at 11:45 AM Tom McDermott via TangerineSDR <
>> tangerinesdr at lists.tapr.org> wrote:
>>
>> Hi Steve - thanks for the offer !
>>
>>
>>
>> This brings up a general question:  should TangerineSDR / PSWS host this
>> kind of software someplace?
>>
>> One could imagine that over time there may be an increasing number of
>> software utilities and packages of
>>
>> interest to the group, it would be nice to have them accessible through
>> one link or portal (ideally with revision control ! ).
>>
>>
>>
>> Is this something the group should consider?  If so, is github the right
>> solution?, a page of links?, something else?
>>
>>
>>
>> -- Tom, N5EG
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Apr 1, 2020 at 5:57 AM Stephen Roland Kaeppler via TangerineSDR <
>> tangerinesdr at lists.tapr.org> wrote:
>>
>> Hi All-
>>
>> I will add a comment if it is of interest, that I wrote some code a while
>> back in python that directly calculates the magnetic field using the IGRF
>> coefficients and derivatives (useful in a ray tracer).  If you guys want
>> that, I could package it up and send it over to anyone interested.
>>
>> Thanks,
>> Steve
>>
>> ------------------------------------------------------------
>> Stephen R. Kaeppler, Ph.D.
>> Assistant Professor
>> Department of Physics and Astronomy
>> Clemson University
>> Clemson, SC 29634
>> Email: skaeppl at clemson.edu
>> Phone: 864-656-4275
>> Web: http://science.clemson.edu/kaeppler/
>> Amateur Radio Callsign: AD0AE
>> ------------------------------------------------------------
>>
>>
>> --
>> TangerineSDR mailing list
>> TangerineSDR at lists.tapr.org
>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>
>> --
>> TangerineSDR mailing list
>> TangerineSDR at lists.tapr.org
>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>
>>
>>
>>
>> --
>>
>> ----
>> Phil Erickson
>> phil.erickson at gmail.com
>> --
>> TangerineSDR mailing list
>> TangerineSDR at lists.tapr.org
>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>
>
>
> --
> ----
> Phil Erickson
> phil.erickson at gmail.com
> --
> TangerineSDR mailing list
> TangerineSDR at lists.tapr.org
> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>

-- 
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
*The way to get started is to quit talking and begin doing.*
*   Walt Disney*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20200402/7a655ad3/attachment.html>