Limited sampling rate when using SoapySDRServer

[gedger]

This may be more of a question than an issue.

I've set up an sdrplay on a Raspberry Pi v3 and run SoapySDRServer, Pi connected via Ethernet to LAN. When using CubicSDR on my MacPro also connected via Ethernet I can only get 1Mbit sampling rate (sometime 1.5Mbit) before the SoapySDRServer starts issuing loads of USB overflow errors. Looking at the Ethernet transmit rate it's less than 5MBps (40Mbps) second so easily within the 90Mbps LAN capability. The CPU on the Pi is also sitting at around 30% so does't appear to be a CPU limitation. This often causes a freeze to CubicSDR although a stop/start device normally cures it however I have seen a couple of crashes.

Any suggestions whether this a Soapy SDR issue, a Pi problem or a problem with CubicSDR not being able to sink the packets quickly enough?

Thanks in advance

Richard

[cjcliffe]

@gedger I'm not exactly sure of the issue; @guruofquality might have more insight (OSX Max MTU again?) but I'll give it a try myself to see as I've only been using RTL-SDRs through the Pi2 so far.

[guruofquality]

There is usually some verbose printed in the server and client side about the stream endpoint sizes. @gedger can you share that? My concern is that the socket buffer cant be allocated large enough. This makes the windowing very small and its harder to get good throughput.

Theres some notes here about the sysctrl limit on linux: https://github.com/pothosware/SoapyRemote/wiki#remote-window I know that OSX wont let you go as high though. But it might help the server side.

[gedger]

I have 2 logs here one from the SoapySDRServer and one from the output of CubicSDR on the Mac. I went straight into sampling a 2M so the logs are as small as possible.

I'll investigate the sysctl limits....

CubicSDRLog.txt
SoapySDRServerLog.txt

[guruofquality]

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=16 KiB
[INFO] Client side stream connected to [fe80::a10d:c228:56cd:7b76%4]:40301
[INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=16 KiB

So 16 KiB isnt very much buffering at all. About 2 ms if I did my math right. SoapyRemote on OSX is actually setup to use a default socket buffer size of 16 KiB because I think larger may cause issues without some sysctl changes -- and I cant have it crashing by default.

• On the linux side, make sure to expand the default the sysctl limits mentioned here
  • https://github.com/pothosware/SoapyRemote/wiki#remote-window
• On the osx side, can you experiment with increasing kern.ipc.maxsockbuf and net.inet.udp.recvspace (there may be others)?
  • https://lists.apple.com/archives/darwin-kernel/2012/Apr/msg00042.html
• For the stream args, also make sure to set a larger remote:window argument
  • Or recompile with a different value here if you cant set the stream arg
• Keep an eye out for those prints "Configured receiver endpoint", they will tell you if the actually socket buffer size was successfully set.

This is sort of an open issue but ideally we figure out and document the OSX procedure to increase the socket buffer size, and perhaps use a better default.

[cjcliffe]

Shouldn't need to recompile; CubicSDR exposes all the relevant remote stream args I believe:

[gedger]

I've had a bit of an investigation and trial but not much success.

Using sudo sysctl kern.ipc.maxsockbuf=6000000

on the mac allowed allowed 4.9M buffers to be allocated which is a 300 fold increase in buffers

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=4882 KiB
[INFO] Client side stream connected to [fe80::a10d:c228:56cd:7b76%4]:59409
[INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=4882 KiB
Stream MTU: 357
starting readLoop()

But it's made no difference at all...

So I used iPerf3 to investigate the throughput (TP) of the link, this showed a TP of 60Mbps. I also played around with the window size, setting 16k dropped TP to 40Mbps, 128k restored the 60Mbps, going higher didn't improve matters. I was expecting a higher overall throughput so something to investigate.

I was working out the the required TP for 2MHz sampling rate, I was expecting 16bit data to be sent CS16 but I notice that the debug says 357 @ 4bytes so is each sample using 4 bytes rather than 2bytes? If it is 4 bytes then 2M * 4bytes * 8 = 64Mbps which is over my TP limit. I was expecting only 16 bit samples which is obviously half which fits OK. Are 4 bytes used because each sample is a 16 bit I and a 16 bit Q?

[gedger]

OK fixed the slow throughput it was my supposedly 1Gbit switch, swapped this out and the TP went up to 90Mbps which is what I'd expect to a Pi3 limited by it's 100Mbps Ethernet.However, even with this 90Mbps throughput stream still stutters at 2MHz sampling....

[guruofquality]

@gedger Thanks for trying out the window stuff, its good to know the procedure now, and also to have eliminated that as a possibility. The sample size for the sdrplay is complex 16-bit samples so 4 bytes per sample or 64Mbps at 2 Msps. Some devices like RTL use 8-bit complex samples (2 bytes). There may be cause to support the same mode with the sdrplay, but it means truncating some bits.

Have you ever done any testing of the play itself on the pi? Something missing from SoapySDR is a simple rate testing app. I'm curious if there are any usb oddities or rate issues on the play side of things w/ the pi. SoapyRemote is pretty much must a thread that call readStream() and socket write(), and especially if the window is large, it should never block on flow control/backpressure. So it would be great to eliminate some other possible causes.

[gedger]

@guruofquality No problems, I have no GUI on the Pi, so I'll have to look for a command line util, perhaps with a web interface, any suggestions?

Alternatively I'll see if I can swap the Pi3 out for a linux box with a 1Gbps interface to eliminate this aspect.

[gedger]

Having looked at the CPU usage on the Pi I now realise that SoapySDRServer is maxing out 1 core of the CPU, so overall CPU usage is <30% but one core is flat out so this may be causing the lost data? Probably no easy way of getting SoapySDR to use multi-cores (1 core for the USB Device, 1 core for the Server?) so will try and substitute the Pi for something else.

[davehun]

what does lsusb -v look like on the pi ?
are both the sdrplay and the ethernet on the same usbhub ?

On 26 Jun 2016, at 20:21, gedger [email protected] wrote:

Having looked at the CPU usage on the Pi I now realise that SoapySDRServer is maxing out 1 core of the CPU, so overall CPU usage is <30% but one core is flat out so this may be causing the lost data? Probably no easy way of getting SoapySDR to use multi-cores (1 core for the USB Device, 1 core for the Server?) so will try and substitute the Pi for something else.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #379 (comment), or mute the thread https://github.com/notifications/unsubscribe/AAzQK-HwYxT1CI7cmLilzeQWt14sZI6xks5qPtEkgaJpZM4I94X8.

[gedger]

Attached output from lsusb -v.

lsusb-v.txt

No USB hub, the SDRPlay is plugged straight into the Pi3 USB. The sdrplay uses around 120mA so well within the pi3 usb power output. Using built in Ethernet port so they are all on the same internal USB bus, (no option on the Pi). However, there's 480Mbps bandwidth available on this so assuming 50% overheads that leaves 240Mbps which gives the Ethernet and sdrplay 120Mbps each which should be enough?

[davehun]

so the sir and the ethernet are bot on the same (on pi) usbhub (Bus 001).

That might well be a limiting factor

On 27 Jun 2016, at 18:46, gedger [email protected] wrote:

Attached output from lsusb -v.

lsusb-v.txt https://github.com/cjcliffe/CubicSDR/files/335483/lsusb-v.txt
No USB hub, the SDRPlay is plugged straight into the Pi3 USB. The sdrplay uses around 120mA so well within the pi3 usb power output. Using built in Ethernet port so they are all on the same internal USB bus, (no option on the Pi). However, there's 480Mbps bandwidth available on this so assuming 50% overheads that leaves 240Mbps which gives the Ethernet and sdrplay 120Mbps each which should be enough?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub #379 (comment), or mute the thread https://github.com/notifications/unsubscribe/AAzQK409djlyWw-8htS1wBo-imy44vfjks5qQAyLgaJpZM4I94X8.

[cjcliffe]

@gedger is there any change in Pi CPU usage when you disable and enable the Automatic Gain? My AGC code in the SDRPlay module is probably about half of it's CPU usage.

edit: might be able to thread the AGC stage if it helps..

[cjcliffe]

@gedger Also might be worth trying the Wifi on the Pi3; it looks like the BCM43438 doesn't use USB just from doing a quick google search -- might get you some more USB overhead if that's the issue.

[gedger]

Have done some more testing.

Pi3 using wifi, 1m away from AP, Macbook running CubicSDR Ethernet

iPerf provides a throughput of ~50Mbit/s which is less than the 64Mbps required at 2MHz sample rate. However, I tested anyway and everything worked fine at 1.5MHz sampling but broke at 2MHz.

One thing I have noticed is that when working at 1.5MHz sampling the Pi3 is using 1 core at about 30%, as soon as you move to 2MHz and USB errors are reported the CPU usage jumps to 90%.

The auto gain option consumes about 10% CPU at the server, so turning off reduced 1 Core CPU to about 20%, however, error condition still takes CPU to 90%. May be worth sticking on separate thread at some point but probably not a priority

I haven't been able to borrow a Linux box to replace the Pi3 to eliminate it, I have been loaned a OrangePI which at least has the Ethernet port and USB on separate buses but can't get the SDRPlay to work on it, something to do with the USB drivers...

My final test is to build a new image for the Pi3 with GUI and build CubicSDR to confirm the SDRPlay works OK without the server involved. Will report back....

One suggested for improvement would be to add an option for compression between the SoapySDRServer and client, there seems to be more CPU power available than BW. Of course complex samples may not compress than well?

[gedger]

I have now built CubicSDR on the Pi3, I guess peoples view of what's usable is different but I found it to be very slow. The CPU is running at 350%, all 4 cores at around 90%. Interestingly the USB buffer overruns started when I switched from 1.5MHz to 2MHz the same as when trying it remotely but the whole user interface is jerky and laggy so probably not a fair test.

I've run out of ideas for the moment :(

[gedger]

I lied, one last thing I just tried...

I used a 1Gbps USB Ethernet dongle plugged into the Pi3 USB. Bizarrely this works with a 5MHz sampling rate providing you turn off the auto gain BUT it will not work at 2MHz sampling rate? Is there something different about the 2MHz sampling rate?

When you enable the auto gain I think the sending of commands to the sdrplay seems to interrupt the receiving side and causes the USB overflow....

[vsonnier]

This issue is replaced by #607, which will be closed soon as well, in the light of recent fixes in the SoapySDRPlay module impacting the remote operation.
In addition, the present issue has gone stale, with the last entry being almost 2 years old.