[aprssig] APRS protocol replacement ideas: protobuf

[Heikki Hannikainen]

On Sun, 13 Feb 2022, Nick VA3NNW wrote:

> Heikki Hannikainen wrote:
>> A protobuf encoded packet format would be very similar in size than the 
>> existing APRS encodings, if the same data is transmitted.
>
> I love the experiment, and some neat data too!

Isn't it. The 1-line encoder and decoder calls are quite a different 
experience to a developer, when compared to what we have to do now!

> ... However I believe protobuf requires the layers below to be 8-bit-clean, 
> no?   :-/

Absolutely. It would require upgrading the complete iGate infrastructure 
to make such packets pass, and setting up a new protocol for the APRS-IS. 
No more TNC2-based igates, KISS only. It would not be a small project. :) 
Which is why it would make more sense for a new development such as LoRa 
or M17, which are going to have their own new gateway software anyway.

Digipeaters might not need modification. If they pass complete binary 
packet contents like classic non-APRS AX.25 digipeaters did, they would 
just work.


In practice, a lot of stuff happening on APRS already kind of requires a 
little more binary cleanliness than exists in practice, beginning with the 
Mic-E format frequently using a few small byte values outside ASCII range. 
Old aprsd versions out there replace those bytes with spaces, and some 
other software replaces multiple consecutive spaces with a single space. 
Some have stripped them out altogether. The APRS-IS servers (javaprssrv, 
aprsc) take this into account when filtering for duplicates. The decoder 
on aprs.fi compensates for those corruptions too: 
https://github.com/hessu/perl-aprs-fap/blob/master/FAP.pm#L1652

UTF-8 uses plenty of high 8-bit values, and that seems to mostly work well 
(haven't tried with UI-View32 though) and there is a noticeable user base 
doing UTF-8 in messages and packet comment fields around Europe and Asia.

One software even used to send UTF-16 encoded unicode messages at some 
point, which, when someone sent an English message, had 2 bytes per 
character, every other byte being a NUL. At some point old aprsd versions 
(I think?) used to truncate messages at the first NUL byte, and I've seen 
new embedded iGates written for ESP32 chips suffering from the same bug.

One new igate written in Python initially tried to treat all incoming APRS 
packets as if they were (fully) UTF-8 encoded, so it UTF-8 decoded them, 
processed the complete frame as a (multibyte) character string, and then 
did UTF-8 encoding again when sending packets out. This doesn't work so 
well when a packet contains binary byte sequences which do not make sense 
to the UTF-8 decoder.

The appearance of these old bugs in the new implementation got me to write 
this, containing some best practices for iGate developers. Maybe it 
should, with some formatting, go to a new APRS document repository.

https://github.com/hessu/aprsc/blob/master/doc/IGATE-HINTS.md


Speaking of UTF-8 and other internalization:

Japan, China, Malaysia and Thailand are very active in APRS, and there are 
local developers out there doing trackers and devices. Examples of 
Atmel-based custom devices being built in Thailand can be found on the 
APRSTH facebook group: https://www.facebook.com/groups/aprsth/

And they have a lot of traffic too:

https://aprs.fi/#!mt=roadmap&z=8&lat=13.5336&lng=100.3894

In China they have a local APRS map site, since Google Maps doesn't work 
there: http://aprs.cn/

What most of us have seen as a side effect of this is the appearance of 
Chinese APRS hand-held and mobile radios. But it's also a noticeably large 
user and developer community on the APRS network and ecosystem, and it 
seems to be growing, while in other places APRS might be getting smaller 
or not growing.

   - Hessu