<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.gmail-m3144113639267677423msolistparagraph, li.gmail-m3144113639267677423msolistparagraph, div.gmail-m3144113639267677423msolistparagraph
{mso-style-name:gmail-m_3144113639267677423msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Tom – first let me say that I agree that there is a real risk that a cheap SBC will not be able to process the amount of data being envisioned no matter how we
code it or write it. That being said, let’s do some benchmarking where we reduce the number of variables.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">My understanding is the GNURadio has Python under the covers. (Is that true?) – if so, Python, being an interpreted language, may itself place limits on processing
speed. Is it possible that this plays a role in the speeds you have been seeing? I’m going to try and do some benchmarking working directly in C, to remove the variables of Python and GNURadio. Maybe I can compare the throughput of various strategies such
as writing raw data vs HDF5, and also with and without conversion to floating point; hopefully this will provide some guidance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There is still the remaining issue: even if we code and save the data in the fastest possible way, at some point we very well may hit a limit of what we can
do with a cheap SBC , on how many slices we can collect at once. The question is, at what point does it significantly affect the science? From 8 slices X 2 antennas, down to what? 5 slices X 2 antennas? Slower sample rates? I guess we’re going to have to
work thru it, but at least it is working in our favor that computers constantly get faster, so the limitations we have today may go away by 2024….<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-73- Bill<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Tom McDermott <tom.n5eg@gmail.com>
<br>
<b>Sent:</b> Tuesday, May 28, 2019 10:51 AM<br>
<b>To:</b> John Ackermann. N8UR <jra@febo.com><br>
<b>Cc:</b> Nathaniel Frissell via TangerineSDR <tangerinesdr@lists.tapr.org>; Engelke, Bill <bill.engelke@ua.edu><br>
<b>Subject:</b> Re: [TangerineSDR] Functional Specification v 0.3<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Bill,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[ MB = Megabyte, Mb = Megabit ]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The Core I7 system I tested on has a HD throughput of 40 MBytes/sec (numerous different tests and benchmarks).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">It's a SATA-3 drive, I suspect the motherboard may be a significant part of the limit, good SATA-3 HD implementations<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">should achieve 100 MB/sec. However 40MB/s is still way above rate that HDF5 was writing: 4 x 192ksps --> 6.4 MB/sec plus<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">some overhead. As John mentions, an SSD would reduce seek-time issues. If there is extensive seek-time then the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">buffers build, although I did not notice any upstream buffer overflows.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I was using Gnuradio HDF5 driver to write the format, I think John was using the same. I could write 4 x 384k to disk<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">in raw format without any problems (no HDF5 coding). The HDF5 coder was the only difference between the two tests.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The Gnuradio HDF5 block takes floating-point input only.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">So I am skeptical that a low-cost SBC is going to achieve HDF5 coding of the raw data in real time. But we don't need<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to code all the raw data, only the data we send to the central server.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here are some floating point vs. fixed point trade-offs I see:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any signal processing done on the Host will want to be done in floating point format. Floating Point DSP is much much easier to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">implement than fixed point processing. With Fixed, you have to watch for underflow and overflow at most steps as<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">either can destroy the signal. Underflow or Overflow means you need to rescale (the entire sample set: long ago, to far future).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Converting from Float to Fixed has the same challenge, you likely need to rescale. Fixed point was common when Floating Point Logic<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">was too expensive (low cost, low power modem chips, CPUs before they universally had Floating Point processors, etc.).
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The drawback to floating point is that the data enlarges from 6 bytes (3-I and 3-Q) in offset binary to 8 bytes (4-I and 4-Q) in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">single precision floating point, increasing storage size and lengthening transfer time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Raw data without significant processing on the DE could be stored & transferred in Fixed format. If the DE does much processing<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">then Fixed becomes an issue. Gnuradio uses floating point for the math blocks (filtering, FFT, AGC, etc.), the Gnuradio soundcard<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sink, the scope and spectrum displays, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In my opinion, the central server is invariably going to want the data in Floating Point format for subsequent processing. So two options<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">are:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1. Convert to Float on the Host, code to HDF5, then send.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. Minimize Host DSP, HDF5 code unprocessed fixed samples and transfer to the Central server, then the central server has to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">unpack HDF5 and transcode to float.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- Tom, N5EG<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- Tom, N5EG<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, May 28, 2019 at 8:02 AM John Ackermann. N8UR <<a href="mailto:jra@febo.com">jra@febo.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">FWIW, I had no trouble capturing 4 x 384 kHz slices with HDF5 on an older i7 laptop with SSD.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">On May 28, 2019, at 10:58 AM, "Engelke, Bill via TangerineSDR" <<a href="mailto:tangerinesdr@lists.tapr.org" target="_blank">tangerinesdr@lists.tapr.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Tom – let’s talk more about the throughput issue. As I mentioned before, I am working right now on
a way to do some benchmarking to start gauging what is possible. I have a related question for you:</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="gmail-m3144113639267677423msolistparagraph"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">On your quad core I7, have you narrowed down where the bottleneck is? There have been some discussions that a rotating hard drive might not be able to keep up with the data
rate, no matter what format is used. (Something else I am going to test). It is quite possible that the compute time to put the data out as HDF5 might be the smaller part of the time budget, and disk writing the larger.</span><o:p></o:p></p>
<p class="gmail-m3144113639267677423msolistparagraph"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">At today’s prices, a 2 TB or 4 TB SSD is a budget buster, but this might not be the case in 2024.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Any thoughts?</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
I am building a crude prototype to see if I can save multiple channels off a Red Pitaya using HDF5 (Digital RF actually, but it uses HDF5), and compare throughput using both spinning drive on USB3 and SSD. -73- Bill</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Tom McDermott <<a href="mailto:tom.n5eg@gmail.com" target="_blank">tom.n5eg@gmail.com</a>>
<br>
<b>Sent:</b> Sunday, May 26, 2019 9:08 PM<br>
<b>To:</b> TAPR TangerineSDR Modular Software Defined Radio <<a href="mailto:tangerinesdr@lists.tapr.org" target="_blank">tangerinesdr@lists.tapr.org</a>><br>
<b>Cc:</b> Scotty Cowling <<a href="mailto:scotty@tonks.com" target="_blank">scotty@tonks.com</a>>; Engelke, Bill <<a href="mailto:bill.engelke@ua.edu" target="_blank">bill.engelke@ua.edu</a>><br>
<b>Subject:</b> Re: [TangerineSDR] Functional Specification v 0.3</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Bill - thanks for iterating the specification. Here are a few comments on 0.3:<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">There are no paragraph numbers to reference, and the page numbers may change<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">depending on how mark-up is selected by the reader, so I'll reference the Title of the paragraph.<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">General Requirements - Assumptions and Dependencies:<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1. The HD may be able to be reduced to 2TB if the 20GE snapshot<br>
can be made while recording to the ring buffer. 2T SSD drives<br>
are coming down in price faster than 4T.<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Technical Notes<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I am skeptical that a SBC-based host can run HDF5 and keep up<br>
with the received data. My quad Core I7-3740 3.4 GHz can not <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">keep up with 4 x 192k from one antenna (one fourth the DE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">requested throughput). It may be better to run HDF5 only on the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">snapshot that is uploaded to the Central Server. That way it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">doesn't have to run at real-time speed, it only needs to code a small<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">subset of the data and can run at a much slower rate paced by how<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">fast the upload link is.<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Data format. The DE will downconvert and decimate the received
<br>
samples. This will produce 24 bit I and 24 bit Q samples, probably 2's<br>
complement binary. These will need to be converted to single<br>
precision floating point I and floating point Q prior to HDF5 encoding.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
-- Tom, N5EG<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
</div>
</div>
<p> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, May 24, 2019 at 12:40 PM Engelke, Bill via TangerineSDR <<a href="mailto:tangerinesdr@lists.tapr.org" target="_blank">tangerinesdr@lists.tapr.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Scotty - Please see attached, updated to include some of the things discussed at Dayton. Next I will work on the Functional Specifications for the Central Control & Database system.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If anyone would like me to start posting to the TAPR github or somewhere, please just text credentials to my mobile number, below. I can assure everyone that I will not make a
mess of it, having done this before.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">W. D. Engelke (Bill), Asst. Research Engr.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Center for Advanced Public Safety<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Cyber Hall<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The University of Alabama<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tuscaloosa, AL 35487<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Desk: (205) 348-7244<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Mobile: (205) 764-3099<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<br>
TangerineSDR mailing list<br>
<a href="mailto:TangerineSDR@lists.tapr.org" target="_blank">TangerineSDR@lists.tapr.org</a><br>
<a href="http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org" target="_blank">http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<pre>-- <br>TangerineSDR mailing list<br><a href="mailto:TangerineSDR@lists.tapr.org" target="_blank">TangerineSDR@lists.tapr.org</a><br><a href="http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org" target="_blank">http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org</a><o:p></o:p></pre>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>