iperf3 in a high delay, high loss environment

[wetglass]

I am characterizing throughput in a FAN (field area network) for low-power IoT devices. iperf3 does not handle high delay well. I will describe my issue here for your consideration.

Overview of WPAN networking being used in this scenario:
https://en.wikipedia.org/wiki/IEEE_802.15.4a

iperf3 Version: 3.6.2

Hardware (endpoints):
Dev endpoint 1:
Linux xubuntu19 5.0.0-15-generic #16-Ubuntu SMP Mon May 6 17:41:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
VM running on Windows 10

Dev endpoint 2: Linux pi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux
Raspberry PI server

Intermediary nodes:
Cisco IR510 FAN cg-mesh wireless nodes
https://www.cisco.com/c/en/us/support/routers/510-wpan-industrial-router/model.html

iperf3 (3.6.2) on Raspberry Pi uses armhf architecture, no apt-get distro build (only 3.1.3), so I had to manually config/make build.

Bug Report

This is a follow-up to the closed bug seen here:
#753

The real-world case is as such:
A customer, like a utility company, could have various power meters, feeders, etc with no internet/data capable links to rx/tx data that allow the endpoint devices to receive commands, or relay data back to the head-end. Thus, using a distributed wireless mesh, a data network is accomplished and remote devices can be reached, either head-end to node, or node-to-node.

The endpoints may require various throughput depending on the application. Generally, it is expected that latency will be high, loss is imminent, and throughput is low. However, there needs to be characterization of these traits to achieve an expected throughput.

The IR510 node has an internal buffering mechanism for receiving and transmitting data (256KB). On a lossy P2P link with variable attenuation due to trees, buildings, passing cars, etc, jitter can be high, adaptive modulation may compensate to vary data rates. Thus, unlike datacenter-type interconnects with expected latencies and throughputs, data shows up as best-effort, when it can.

iperf3 does not handle this situation well at all. It does handle the scenario when attenuations are moderate, loss < ~70%, throughput > ~1k/s. But this is the bottom end, after which, iperf3 dies in the exact fashion described in bug #753 (linked above).

A simple end-to-end link:
[dev] --- [node] --- [node] --- [node] --- [dev]

Devices would be connected via ethernet RJ45 to the adjacent node, and nodes establish neighbor connections via 900mhz wifi.

As noted above, the importance of the scenario is not how much throughput happens in what time, but simply what the capable overall throughput is.

This is my suggestion for a solution:
I think a pulse calculation of frames received would solve this FAN throughput issue. I imagine a data stream would have to be considered a "data pulse", and throughput calculated based on not just length of time between first/last rx, but also average pulse throughput of received frames (not datagrams). Basically, instead of measuring layer 4 throughput, we're measuring layer 2 throughput and upscaling into a layer 4 throughput interpretation subtracting delays between frames. Precision is arbitrary due to upscaling. I don't need precise, relative is just fine.

I'd like to hear ideas on this solution, as well as take some time to think about how to improve it.

If anyone reading this is familiar with delay-tolerant networking, or even interplanetary internet, one can see the benefits of being able to characterize throughput in a hostile/unpredictable network environment. The answer may even be found there.
https://en.wikipedia.org/wiki/Delay-tolerant_networking
https://en.wikipedia.org/wiki/Interplanetary_Internet

[wetglass]

I have created a paste to demonstrate the behavior described:
http://oneclickpaste.com/14865/

[acooks]

@wetglass, it sounds like you're asking for a completely different approach to the problem of capacity estimation. What you're describing sounds a bit like the pathChirp approach. Have a look at http://www.spin.rice.edu/Software/pathChirp/

On the other hand, I can report that I've seen greatly improved behaviour on high RTT/ variable RTT networks with the changes in PR #859.

[bmah888]

@wetglass: I think I agree with @acooks...you're trying to solve a different problem than what iperf3 was designed to do. iperf3 was designed for testing the performance of bulk transfers using TCP and UDP, in typical R&E network environments (which are typically high-bandwidth, wired networks). In lossy environments, these protocols (TCP in particular) won't perform well. This isn't surprising, right?

You might be able to learn something by doing UDP tests in iperf3, but I think a completely different tool is needed for what you're trying to measure.

[wetglass]

I don't disagree @acooks in that iperf may not be the best tool for the job, though it has been useful for many tests so far. You're correct @bmah888 in that either TCP or UDP is not expected to perform well. Just using say VNC on this type of link is a chore!

I've been looking into owping per your suggestion @bmah888 in #532. There's some internal tweaking due to NAT behavior on these nodes I have to resolve first before this succeeds.

Thank you for the suggestion @acooks about pathChirp. I'll follow up on that to see if it fits my testing. I'm interested in how PR #859 may improve my situation!

I'll also try nuttcp though I'm not sure how it will handle these sort of delays. Saw this comparison chart, and the note that nuttcp is likely for high throughput networks, which mine is very much not.

[acooks]

@wetglass, in case you missed it, there's a 3.7 release. It may improve the behaviour you saw, so if you can, please try it.

[wetglass]

I'll definitely try it out and let the thread know the results.

[wetglass]

Impressions:

1. No longer hangs/dies in high loss/delays scenarios. Awesome.
2. Server "gets lost" when testing in abysmal network conditions, where the connection remains open and it's expecting more packets, thus remains in an endless state.
3. iperf3 -v still shows 3.6
4. No change in throughput in said scenarios, but wasn't expecting any.
5. Jitter still shows only last value, is this supposed to be averaged?

Very happy #1 is fixed! 👍

[bmah888]

Glad this is working better...I’m taking a little time away from work (and yes this is work for me), so I just wanted to reply to item 3 above...I just rebuilt iperf3 from source on a server and the version string does correctly show 3.7 for me. That’s probably the least important of the points you made above, but the only one I can easily check at the moment. If I remember right the jitter computation is a kind of weighted average (according to an RFC that’s cited in the source code).

[wetglass]

Huh, that's strange. I physically moved the binary away from /usr/local/bin on one of my machines, and apt-get remove'd on another, still showing 3.6. On jitter, it always uses the last iterated value.

Thank you for your efforts in maintaining the code!

[bmah888]

It sounds like this issue is resolved, so closing.