[TangerineSDR] Update on testing RTLSDR and KA9Q-radio

[Tom McDermott]

After a lot more experiments, here's an update on the multicast packet drop
issue when using ka9q-radio.

The multicast RTP stream rate is about 12 Mbits/sec (192 ksps IQ one RF
channel).  The client RTP
software replaces the data from a dropped packet with the (apparently)
correct number of zero-value samples when
it recognizes a dropped packet. This prevents wild excursions in the client
received data rate.

The Raspberry Pi 4 was replaced with a core i5-11500T running Ubuntu 22.04
(much more powerful).
This did not change the 0.1% packet loss rate issue.   However it did solve
the problem of the R Pi 4 not
being able to talk very well to the client core i7 (sink) with a directly
wired Ethernet connection.
The i5-11500T runs clean to the client with iperf3 both directions under
all conditions tested
(no packets lost for big bursts at line rate (about 940 Mbits/sec). It
should be noted
that this iperf3 test uses TCP to IPv4 LAN addresses (i.e. 192.168.x.x).

WIth a hard wired direct connection between the client and server there is
no multicast packet loss.
When using either of two Ethernet switches the loss rate is ~0.1% (with
statistical variation).
This is at a net data rate of 12 Mbits/sec. All the links are GbE.

When source and sink are on the same computer it works without packet loss
(via 127.0.0.1).

Based on limited research it appears that packets in the multicast address
range might be considered
exception packets by many switches. These packets are diverted from the
switch's fast-path (hardware)
to the slow-path (software) for inspection in case they are IGMP packets,
even if they are not.
The switches lose ~0.1% regardless of whether IGMP is enabled or disabled.
If the (relatively slow)
switch processor can't keep up, it likely discards the packet.

The switches were tested with only 2 ports UP, all the other ports
physically disconnected (DOWN).
This ensures that there are no WiFi links anywhere connected, and that
there is no other management
traffic traversing the switch, and IGMP doesn't have to deal with flooding
congestion (since one does not flood
a DOWN link).

The source is not dropping packets according to tshark when the switch is
used, but the sink is
dropping them.

-- Tom, N5EG






On Tue, Sep 12, 2023 at 3:49 PM Tom McDermott <tom.n5eg at gmail.com> wrote:

> Hi Phil - unfortunately there's no standard for Jumbo frame size amongst
> the manufacturers.
> While they can go up to 16k in size, support is usually widespread for
> 9000 bytes, fair at 9600 bytes, and
> degrades from there. They are especially helpful starting at 10 GbE.
> Routers benefit the most from
> Jumbo frames as it reduces the number of packets per second that have to
> lookup next-hop egress
> ports, and that is mostly a software function.
>
> The KA9Q software right now is sending multiple lower bit rate streams and
> I think (without looking
> through the code) doesn't benefit from the larger frames. I don't know
> what Phil Karm might be thinking about there,
> he would be the expert to comment.
>
> The Tangerine modules are on hold right now due to various issues.  The
> Clementine modules are intended to
> feed into the KA9Q software.
>
> -- Tom, N5EG
>
>
> On Tue, Sep 12, 2023 at 2:08 PM Phil Erickson <phil.erickson at gmail.com>
> wrote:
>
>> Hi Tom,
>>
>>   That's a lot of excellent work.  Curiosity: at Haystack, we often find
>> we have to fiddle with network switch settings to handle 'jumbo frames'
>> (MTU > 1500) right.  This comes up as you increase the sampling rate from a
>> particular software radio and is of course totally dependent on the wire
>> frame protocol between the SDR and driver.  Do you expect any of the
>> Tangerine radios to produce jumbo packets, and in any case, does the RPi4
>> handle that case OK without dropping things?
>>
>>   Or perhaps you never get to this use case?
>>
>> 73
>> Phil W1PJE
>>
>> On Tue, Sep 12, 2023 at 3:36 PM Tom McDermott via TangerineSDR <
>> tangerinesdr at lists.tapr.org> wrote:
>>
>>> WIth awesome support from Franco, K4VZ, there has been a lot of progress
>>> in testing
>>> KA9Q software over a wired home network.
>>>
>>> The test setup uses an RTLSDR dongle to send 192 ksps IQ data from a
>>> Raspberry Pi to
>>> an Ubuntu 22.04 core i7 host running gnuradio with Franco's gr-rtp
>>> module. This test
>>> setup may be one where folks have available hardware available from the
>>> junkbox and
>>> thus able to validate their system.
>>>
>>> The FFTW wisdom file was computed on each system before starting the
>>> tests. All
>>> connections are using 1 GbE wired connections, no WiFi links in the
>>> system. The GbE
>>> switch is configured with IGMP snooping turned on to avoid multicasting
>>> traffic to
>>> unsubscribed ports. IGMP was verified as working.
>>>
>>> Initially a RPi 3B+ with GbE interface was used. At 192 ksps It had a
>>> packet drop rate of over 10 %
>>> resulting in unusable performance. The gnuradio flowgraph was not able
>>> to demodulate
>>> audio from the IQ stream.  It did successfully demodulate audio when the
>>> test was run
>>> at 48 ksps, with lower packet loss rate.
>>>
>>> Changing to a Raspberry Pi 4 at 192 ksps improved the drop rate to
>>> 0.4%.  This allowed gnuradio
>>> to demodulate audio, but with some artifacts.
>>>
>>> Last night the fft-threads=4  line was commented out of the radiod conf
>>> file, allowing
>>> ka9q-radio to use its default of 2 threads for the fft.  This further
>>> dropped to packet loss
>>> rate to 0.1%.  The packet losses are now infrequent (perhaps once every
>>> 15 seconds).
>>>
>>> Separately all the computers and ethernet connections have been tested
>>> with iperf3.
>>> This is a multi-platform tool (originally Unix / Linux) that sends
>>> bursts of data between the endpoints
>>> measuring the throughput and the TCP retry rate.
>>>
>>> The Ubiquiti switch does have retries, sometimes a low number, sometimes
>>> a bit larger number.
>>> A separate small HW switch was substituted and it has zero retries with
>>> iperf3.  All the
>>> tests show a throughput of about 940 Mbit/s regardless, so the retry
>>> rates seem pretty small.
>>>
>>> The ka9q-radio packet drop rate is the same (0.1%) using either the test
>>> with the Ubiquiti switch
>>> or the small HW switch, so it doesn't seem to make much difference.
>>>
>>> The drop rate was tested with gnuradio running, and with gnuradio not
>>> running. The latter uses
>>> pcmcat {addr/ssrc} > /dev/null to keep the IGMP path open , and the loss
>>> rate is measured with a
>>> tshark script provided by Franco.
>>>
>>> The test was repeated with ka9q-radio source and gnuradio both on a
>>> single core i7 computer
>>> (using the loopback data connection). That system runs without drops for
>>> a few minutes
>>> (which was the limit of the test duration).
>>>
>>> Franco has tested RPi4 to Linux host with a direct connection without
>>> drops. However he is
>>> using a different SDR receiver than RTLSDR for that test.
>>>
>>> A separate test was run with a direct hard wired connection from the
>>> RPI4 to the Linux host.
>>> One has to set static IP addresses, and inject static IP routes for the
>>> LAN and the multicast
>>> range 224.0.0.0/4 on both ends.   That setup was unusable with about
>>> 50% packet loss rate,
>>> but this was prior to the fft-threads change (above). It has not been
>>> rerun since the fft-threads change.
>>>
>>> Some speculations and observations:
>>>
>>> * The different SDR drivers appear to have some significant performance
>>> variations.
>>>
>>> * Different switches and links have different iperf3 retry rates, but
>>> they don't seem to affect the
>>> drop rates much.
>>>
>>> * The RPI3 B+seems a bit too slow for 192 ksps, but was marginally OK
>>> for 48 ksps.
>>>
>>> * A faster SBC might be beneficial in place of the RPI4. That would be a
>>> good test to try.
>>>
>>> Again, much thanks to Franco, who has provided software instrumentation
>>> to measure the loss rates
>>> both inside of gnuradio and outside of gnuradio (i.e. with gnuradio not
>>> running).
>>>
>>> -- Tom, N5EG
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> TangerineSDR mailing list
>>> TangerineSDR at lists.tapr.org
>>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>>
>>
>>
>> --
>> ----
>> Phil Erickson
>> phil.erickson at gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20230920/fa20673c/attachment.html>