[TangerineSDR] Benchmarking Digital RF (HDF5)

[Tom McDermott]

Hi Phil - there are consumer HD that are designed for video surveillance
systems and are
specified for continuous duty writes.  Western Digital calls them
'Purple'.  Seagate calls
them Skyhawk Surveillance.  The pricing is reasonable,  4T for about $110
on Amazon.

WD Purple specifies the 4T to handle 180 TB write / year.   If PSWS writes
2 T / day then we
operate at 730 TB/ year.    That's PSWS using 2 antennas, 8 RF channels,
192 ks/s.
We may operate less on average depending on the science campaign.

I don't know whether the HD will actually handle that workload, or if it's
just a warranty play.

-- Tom, N5EG



On Wed, May 29, 2019 at 7:44 AM Phil Erickson via TangerineSDR <
tangerinesdr at lists.tapr.org> wrote:

> Hi Bill,
>
>   FYI, our software radar at MIT Haystack (Ettus USRP N210 hardware as the
> source; GbE interface, running 25 complex megasamples / second at 16 bit
> depth) goes to a recorder agent that formats to Digital RF.  This streams
> about 191 MB/sec of HDF5 files produced by the Digital RF API to a ring
> buffer inside a ramdisk, so going directly to memory.  A subsequent agent
> processes power spectra with heavy reduction to a conventional spinning
> disk set (RAID 6, I believe) with a GlusterFS** cloud overlaid.  This all
> uses workstation/general purpose computing on Ubuntu platforms.  Note that
> we directly run a cable between the USRP and the computer and do not have
> it pass through any switches/routers.
>
>   So I guess I don't have a data point because our RAM disk buffers any
> slowdowns until we get it to spinning media.  We use a ~100 GB ram buffer
> which provides probably 230 seconds of realtime soak per channel (x 2
> channels for us).  I know that we tried writing the 191 MB/second to
> spinning disks and it does not work - one needs SSDs at that rate.
> Probably worth trying SSDs in your testing.
>
>   Aside: we do find that writing continuously to spinning media makes it
> die faster than normal since the manufacturers do not guarantee performance
> with continual write unless you buy Enterprise/server grade hardware  - 10x
> more expensive, naturally.  This is in line with Google's server farm
> experience using JBOD or equivalent systems.
>
> 73
> Phil W1PJE
>
> ** "GlusterFS is a scalable network filesystem suitable for data-intensive
> tasks such as cloud storage and media streaming."  https://www.gluster.org
>
> On Wed, May 29, 2019 at 10:08 AM Engelke, Bill via TangerineSDR <
> tangerinesdr at lists.tapr.org> wrote:
>
>> Benchmarking Digital_rf on Odroid
>>
>>
>>
>> There have been some deep concerns about the ability of a SBC to keep up
>> with saving spectrum data using HDF5 (here using the MIT Digital_rf
>> package, which uses HDF5 internally). I installed Digital_rf on my Odroid
>> XU4 and ran the benchmarks that are included in the Examples folder.
>>
>>
>>
>> Writing to MicroSd card in the XU4 (SanDisk Ultra Plus, claimed speed 80
>> MB/sec), benchmark-reported speeds
>>
>> 1.                     Writing raw binary data directly
>>                                    133.36  MB/sec
>>
>> 2.                     Writing “simple single write” in HDF5
>>                                 154.14 MB/sec
>>
>> 3.                     Writing “simple single write” in HDF5 with
>> checksum        71.29 MB/sec
>>
>> 4.                     Writing      “        “    “  with checksum +
>> compression L1        8.01 MB/sec
>>
>> Unexpected results here:
>>
>> a. HDF5 "simple write" is faster than writing raw binary
>>
>> b. top two speeds are faster than the claimed speed of the MicroSD card.
>> Maybe the benchmark is measuring a burst rate during just the HDF5
>> processing (?) Maybe there is some caching  or buffering going on.
>>
>>
>>
>> Writing to Western Digital USB 3 (spinning) external 4TB hard drive
>> (incl. 64 MB internal cache) – benchmark-reported speeds
>>
>> 1.                     Writing raw binary data directly
>>                                    103.66  MB/sec
>>
>> 2.                     Writing “simple single write” in HDF5
>>                          96.04 MB/sec
>>
>> 3.                     Writing “simple single write” in HDF5 with
>> checksum          55.02 MB/sec
>>
>> 4.                     Writing      “        “    “  with checksum +
>> compression L1        7.67 MB/sec
>>
>> These results make a bit more sense than the MicroSD result. I can't find
>> a claimed transfer speed for this hard drive on Western Digital's site.
>>
>>
>>
>> IF these results are representative (a big “if”) – then even a 4 TB
>> spinning hard drive will be able to keep up with the PSWS data write rate.
>> If anyone can confirm, reproduce, add to, or refute these results, I am
>> most interested to know…
>>
>>
>>
>> -73-
>>
>> W. D. Engelke (Bill), Asst. Research Engr. AB4EJ
>>
>> Center for Advanced Public Safety
>>
>> Cyber Hall
>>
>> The University of Alabama
>>
>> Tuscaloosa, AL 35487
>>
>> Desk: (205) 348-7244
>>
>> Mobile: (205) 764-3099
>>
>>
>> --
>> TangerineSDR mailing list
>> TangerineSDR at lists.tapr.org
>> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>>
>
>
> --
> ----
> Phil Erickson
> phil.erickson at gmail.com
> --
> TangerineSDR mailing list
> TangerineSDR at lists.tapr.org
> http://lists.tapr.org/mailman/listinfo/tangerinesdr_lists.tapr.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tapr.org/pipermail/tangerinesdr_lists.tapr.org/attachments/20190529/50c9f433/attachment-0001.html>