[aprssig] Unicode in APRS messages

[Matti Aarnio]

On Sun, May 11, 2014 at 08:10:19AM +0200, SARTrack Admin wrote:
> I am currently adding internationalisation to my SARTrack program (
> http://www.sartrack.co.nz ), and have volunteers in various
> countries doing translations for me.
> 
> Up to this point, I never attempted to use Unicode for APRS
> messaging during transfer via APRS servers.
> 
> I have now added UTF8 encoding/decoding to the TCP interface... And
> this works perfectly when I use my own program, while testing on a
> 'aprsc' based server. Russian and other languages with 'funny'
> characters now come out correctly on the other side.
> 
> However, when I tested it with the APRS client 'AGWTracker', garbage
> came out.

That is to be expected.  AGWTracker does support UNICODE, but it uses
UCS-2 or UTF-16 encoding instead of UTF-8 depending on Windows version.

  http://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows

I can even understand (though not approve) AGW's reasoning behind it:

  Greek and Asian character are two bytes each, instead of 2 or 3 or more.

Unfortunately both UCS-2 and UTF-16 does introduce with ASCII and ISO-8859-1
character codepoint subsets within UNICODE a situation where every second
byte is 0x00, and that is really troublesome with many softwares. Not to
ignore that it also produces excessively large encoding for ASCII users.

Also it applies directly to native Windows display API without need to
do conversions (a very stupid reason for Windows programs, while maybe
reasonable for embedded systems.)

ASCII is UNICODE subset, and for UTF-8 encoding the byte stream is
identical with ASCII in 8-bit encoding. No spurious 0x00 bytes.


> I would like to know how other APRS clients deal with this issue,
> and what Encoding (if any) is being used to transfer Unicode APRS
> packets over TCPIP.
> And how this type of data is dealt with, when it must be transfered
> over the radio, as this is obviousely ASCII/ANSI based...
> 
> Thanks,
> Bart ZL4FOX / PA2FOX

73 de Matti, OH2MQK