[aprssig] distributed aprs visualization possible ?

[Michael Conrad]

Hi all,

> See my last reply for some findU stats. Keep in mind what you are  
> talking about here is cherry picking the easy stuff findU does. This  
> is what aprs.fi does, and to some extent aprsworld, but you can't call  
> it a findU replacement unless it does the hard stuff. What are you  
> plans for handling:
okay, I renamed the thread to a distributed aprs visualization with the
primary focus on a short-term visualization, it should not be an aprs
archive.

@Steve: Do you have some statistics, how often a specific function (position report, wx report, history, ...) is used on 
findu ?

I think the most used function today is something like: what is the current position/status
of station X and which other stations are nearby of station X.

Therefore the primary function of a (distributed) aprs visualization is a realtime
overview of the status of a given station and information in an area of interest around this station.

> How are you going to show month+ long tracks?
in best case not at all ;-)

For a long term storage, you need more storage and better replication. Eventually this can be added later,
but for the first step a short-term visualization should be enough challenging.

> With your distributed system, how do you handle a guy that travels  
> from the area covered by one server to another? There are lots of  
> details you need to address...
okay, I try to provide some technical details, but it is only an idea

Disclaimer: This is no finished design, it is only an idea and I want to discuss, it is possible or not.

What such a system should provide: distributed short-term visualisation of aprs traffic (position, status, nearby stations)

What such a system should not provide: long-term storage of aprs data, global aprs archive,
                                        replacement for aprs-is (not yet ;-)

I identified following key problems of a distributed aprs visualisation:

- Data partitioning (Which peer is responsible for a given area?)
- Data lookup (Which peer stores data about station X?)
- User interaction (How a user could use the distributed system?)


Data partioning
===============

Requirements: acquisition of all aprs data and complete coverage, reorganisation if peers fail or going offline

Solution: All peers are organized in a CAN overlay (http://www.sigcomm.org/sigcomm2001/p13-ratnasamy.pdf). CAN stands
           for "Content Addressable Network". The key idea behind CAN is to have a d-dimensional cartesian coordinate
           space. In our example I would prefer an 2-dimensional coordinate space like the WGS84 coordinate  system
           (x: -180 - 180, y: -90 - 90).

           In a CAN overlay each peer is responsible for rectangle described by coordinates
           (i.e. 49.00, 8.00, 50.00, 9.00, somewhere around Karlsruhe, Germany). If a new peer is joining
           the CAN overlay, he choose a coordinate (his home location) and contacts the peer, which is responsible
           for the area (now called zone), where the own coordinates are located in (for routing, please read paper).
           Now, the existing zone will be splitted up in two zones, each peer is responsible for a half of the
           old zone.

           CAN is an peer-to-peer network. Except bootstrapping, no central instance is needed. CAN provides
           reorganisation in case of peer failures and is able to handle a high number of peers. Each peer only
           need to handle a small number of informations (ip addresses of neighbour peers).

           After joining the CAN overlay, each peer connects an APRS-IS server and requests all APRS data for a given
           area (using a-filter). This area is identically with the zone in the CAN overlay. In case of a reorganisation
           of the CAN overlay (failure of neighbour peer, new peer join) the peer will adapt the size of the area of
           interest. Each peer stores all aprs messages coming from the APRS-IS server locally.

Data lookup
===========

Requirements: localisation of the responsible peer, who stores data about station X

Solution: In addition to the CAN overlay, each peer is part of a distributed hash table (DHT)
           (http://en.wikipedia.org/wiki/Distributed_hash_table). A distributed hash table is a distributed database
           above a structured overlay network (i.e. Chord, Kademlia (known from Bittorrent), Bambo, ...).
           Normally distributed hash tables are able to store/retrieve a abitrary value under a given key (put/get).
           A public example of a distributed hash table in the wild is OpenDHT, please visit: www.opendht.org for
           more information.

           If a peer receives an aprs message from the APRS-IS server, he stores his own ip address under the callsign
           of the station (put ABCDEF 1.1.1.4). This is only necessary, if the peer receives the first aprs message from
           a station, otherwise this part can be skiped.

           To know, which peer is responsible (or other question: Where is last known position of station ABCDEF),
           another peer can query the distributed hashtable for ABCDEF (get ABCDEF -> result 1.1.1.4).

User interaction
================

Requirements: simple user interface, transparent migration to responsible peer.

Solution: If a user requests www.distributedaprsvisualisation.org a DNS load balancing will be used to distribute
           all users across all active peers. Each peer is running a small web server (lighttpd) with a (full) functional
           web application to show station's position/status/ADD_YOUR_FEATURE_HERE, but only with a limited data
           repository.

           After the user enters the callsign of a station, the peer requests the ip address of the responsible peer
           and uses a HTTP redirect to move the user to the responsible peer. The responsible peer queries the local
           data repository and sending some javascript (to requests the maps from google maps/yahoo/osm) and the
           information about the requested callsign (or list of callsign if you want to show all nearby stations)
           back to the user.


Additional disclaimer: This is only a design draft, but I am interested in discussing, if such a system is feasible.
                        If not, I have wasted some time, but I am willing to invest this time.


Have a nice sunday

      Michael

PS: Congratulations, you have reached the end of this mail ;-)