High UDP packet loss, compare with nuttcp

[IlorDash]

Context

• Version of iperf3: iperf 3.16 on Server and iperf 3.5 on Client

• Hardware: Receiver - aarch64 Quad-Core ARM Cortex-A53 CPU, Sender - x86_64 Intel Core i5-11600KF with NIC Intel I219-V

• Operating system (and distribution, if any): Server - Buildroot 2023.05 Linux 5.10, Client - CentOS Stream 8 Linux 4.18

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.): Server and Client are connected straight with CAT 5E Ethernet cable (without any routers between them)

I'm running UDP tests with iperf3 and nuttcp-6.1.2 on 1Gb net and getting big difference in datagrams loss. With iperf3 27% loss and with nuttcp nearly zero loss. iperf3:

$ iperf3 -t 10 -u -b0 -c 192.168.1.2
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   949 Mbits/sec  0.000 ms  0/819410 (0%)  sender
[  5]   0.00-10.00  sec   828 MBytes   695 Mbits/sec  0.027 ms  219702/819277 (27%)  receiver

# iperf3 -4s
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   828 MBytes   695 Mbits/sec  0.027 ms  219702/819277 (27%)  receiver

nuttcp:

$ nuttcp -v -T10 -u -Ru -i1 192.168.1.2
nuttcp-t: v8.1.4: socket
nuttcp-t: buflen=1024, nstream=1, port=5101 udp -> 192.168.1.2
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: send window size = 212992, receive window size = 212992
nuttcp-t: available send window = 106496, available receive window = 106496
nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=1024, nstream=1, port=5101 udp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: send window size = 212992, receive window size = 212992
nuttcp-r: available send window = 106496, available receive window = 106496
  106.2783 MB /   1.00 sec =  891.4452 Mbps  4690 / 113519 ~drop/pkt  4.13 ~%loss
  108.9092 MB /   1.00 sec =  913.6412 Mbps  2055 / 113578 ~drop/pkt  1.81 ~%loss
...
  109.0479 MB /   1.00 sec =  914.7633 Mbps  1913 / 113578 ~drop/pkt  1.68 ~%loss
nuttcp-t: 1109.2139 MB in 10.00 real seconds = 113583.38 KB/sec = 930.4750 Mbps
nuttcp-t: 1135842 I/O calls, msec/call = 0.01, calls/sec = 113584.08
nuttcp-t: 0.0user 1.2sys 0:10real 13% 0i+0d 1638maxrss 0+1pf 21405+5csw

nuttcp-r: 1088.2109 MB in 10.00 real seconds = 111419.31 KB/sec = 912.7470 Mbps
nuttcp-r: 21507 / 1135835 drop/pkt 1.89% data loss
nuttcp-r: 1114332 I/O calls, msec/call = 0.01, calls/sec = 111419.71
nuttcp-r: 1.5user 7.7sys 0:10real 89% 0i+0d 872maxrss 0+2pf 17183+44csw

# nuttcp -S -4 --nofork

Dropwatch shows drops in udp_queue_rcv_one_skb during iperf3 tests:

22253 drops at udp_queue_rcv_one_skb+3bc (0xffffffc01096521c) [software]

There are a lot of UDP InErrors and RcvbufErrors in /proc/net/snmp :

...
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
Udp: 1720917 3 212177 178 212177 0 0 79
...

So, I think UDP drops here.

Overhead comparison, iperf3:

Samples: 40K of event 'cpu-clock', Event count (approx.): 10010750000                                                                                                                                             
  Overhead  Command  Shared Object          Symbol                                                                                                                                                                
+   15.08%  iperf3   [kernel.kallsyms]      [k] remove_wait_queue                                                                                                                                                 
+   14.58%  iperf3   [kernel.kallsyms]      [k] add_wait_queue                                                                                                                                                    
+   11.02%  iperf3   [kernel.kallsyms]      [k] do_el0_svc                                                                                                                                                        
+    7.11%  iperf3   [kernel.kallsyms]      [k] do_csum                                                                                                                                                           
+    6.27%  iperf3   libc.so.6              [.] __select                                                                                                                                                          
+    6.10%  iperf3   libc.so.6              [.] read                                                                                                                                                              

nuttcp:

Samples: 39K of event 'cpu-clock', Event count (approx.): 9830250000                                                                                                                                              
  Overhead  Command  Shared Object          Symbol                                                                                                                                                                
+   16.18%  nuttcp   [kernel.kallsyms]      [k] do_csum                                                                                                                                                           
+   15.03%  nuttcp   [kernel.kallsyms]      [k] do_el0_svc                                                                                                                                                        
+   14.90%  nuttcp   libc.so.6              [.] recvfrom                                                                                                                                                          
+   13.54%  nuttcp   [kernel.kallsyms]      [k] __local_bh_enable_ip                                                                                                                                              
+    8.53%  nuttcp   [kernel.kallsyms]      [k] arch_local_irq_restore                                                                                                                                            
+    4.14%  nuttcp   [kernel.kallsyms]      [k] __arch_copy_to_user                                                                                                                                               
+    1.82%  nuttcp   [kernel.kallsyms]      [k] udp_recvmsg                                                                                                                                                       

Since select in iperf3 has a lot of overhead, as an experiment I tried to remove all selects during read in src/net.c and get nearly zero packet loss:

# iperf3 -4s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.10 GBytes   948 Mbits/sec  0.013 ms  713/819070 (0.087%)  receiver

There are no such problems with TCP with both utilities.
I'm searching for any help to understand reasons of packet loss difference between iperf3 and nuttcp. I've also found suggestion to use nuttcp for high-speed UDP testing at fasterdata, but there are no reasons for doing that.

[bmah888]

You might want to try increasing the socket buffer size(s), using the -w command-line flag. (It looks like the flag sets the TCP window size, but it does that indirectly by setting the socket buffer size, and that's applicable to UDP tests as well.

iperf3 does a lot of select() calls. One of the reasons for that is to implement bounds on how long tests are supposed to run. We could consider doing that some other way.

[IlorDash]

Thank you for your reply! I attempted to increase the receive buffer using the -w option, but it didn't resolve the issue. I can confirm that the buffer size increased, as I observed the rx_queue filling up in /proc/net/udp. Additionally, increasing the buffer size slightly improved the number of bytes received with nmap netcat. Furthermore, the -w flag indeed sets the receive buffer size, as verified through strace iperf3 ... -u -w 425984 -R:

setsockopt(5, SOL_SOCKET, SO_RCVBUF, [425984], 4) = 0
setsockopt(5, SOL_SOCKET, SO_SNDBUF, [425984], 4) = 0
getsockopt(5, SOL_SOCKET, SO_SNDBUF, [425984], [4]) = 0
getsockopt(5, SOL_SOCKET, SO_RCVBUF, [425984], [4]) = 0

My goal is to confirm (or deny) that my SoC can achieve 1 Gb/s in UDP use cases, such as video streaming.

I also forgot to mention that my SoC has a Quad-Core ARM Cortex-A53 CPU.

[davidBar-On]

I was not able to recreate the issue is my environment (single computer), but while trying to do that I found a problem in the iperf3 server, that cause it to use almost twice the CPU really needed for a single stream UDP test. I fixed the issue in PR #1708. As the CPU overhead may cause loosing UDP packets, can you try building and testing using the PR code, to see if the reduced CPU overhead helps?

[IlorDash]

Thank you for your help! I tested iperf3 with your PR, but unfortunately, it didn't help. The packet loss is still around 27%, and the overhead remains the same.

[davidBar-On]

Thanks for trying. Just to make sure, did you use the PR version for both the client and the server?

[IlorDash]

Yes, I built iperf3.16 with your PR for both the client and server.

[IlorDash]

I tried running iperf3 with two streams, which showed zero drops:

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-120.05 sec  6.68 GBytes   478 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-120.00 sec  6.68 GBytes   478 Mbits/sec  0.034 ms  2303/4955912 (0.046%)  receiver
[  7]   0.00-120.05 sec  6.68 GBytes   478 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  7]   0.00-120.00 sec  6.68 GBytes   478 Mbits/sec  0.032 ms  2244/4955901 (0.045%)  receiver
[SUM]   0.00-120.05 sec  13.4 GBytes   956 Mbits/sec  0.000 ms  0/0 (0%)  sender
[SUM]   0.00-120.00 sec  13.4 GBytes   956 Mbits/sec  0.033 ms  4547/9911813 (0.046%)  receiver

With two streams, iperf3 utilizes two CPUs (I monitored this with mpstat -P ALL 1 and the verbose output of iperf3), but the overhead remains the same. I also noticed differences between running in Server and Reverse modes. In Server mode, iperf3 uses CPU 2 and CPU 3:

18:40:43     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
18:40:44     all    6.27    0.00   43.86    0.00    1.75   23.31    0.00    0.00   24.81
18:40:44       0    0.00    0.00    0.00    0.00    7.00   93.00    0.00    0.00    0.00
18:40:44       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
18:40:44       2   12.00    0.00   88.00    0.00    0.00    0.00    0.00    0.00    0.00
18:40:44       3   13.00    0.00   87.00    0.00    0.00    0.00    0.00    0.00    0.00

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.00  sec  2.40 GBytes   686 Mbits/sec  0.020 ms  700725/2476903 (28%)  receiver

In Reverse mode, iperf3 uses only CPU 3:

18:41:35     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
18:41:36     all    4.00    0.00   21.00    0.00    2.00   23.00    0.00    0.00   50.00
18:41:36       0    0.00    0.00    0.00    0.00    7.07   92.93    0.00    0.00    0.00
18:41:36       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
18:41:36       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
18:41:36       3   15.84    0.00   83.17    0.00    0.99    0.00    0.00    0.00    0.00

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.04  sec  3.35 GBytes   957 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-30.00  sec  2.21 GBytes   632 Mbits/sec  0.020 ms  843100/2481017 (34%)  receiver

As a result, there is a slight difference in bandwidth between these two modes.

Additionally, I compared iperf3.16 with iperf3.13, which is default in buildroot 2023.05 that I'm using. The earlier version shows worse bandwidth:

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-120.05 sec  13.4 GBytes   956 Mbits/sec  0.000 ms  0/9911810 (0%)  sender
[  5]   0.00-120.00 sec  7.50 GBytes   537 Mbits/sec  0.027 ms  4351830/9911494 (44%)  receiver

At this moment, I see two options to achieve better performance results in my case:

• Use iperf3.16 or later versions;
• Run iperf3 with two streams.

[zhlhahaha]

Hi @IlorDash,
I used to encounter the same issue. I suspect that the multiprocess iperf3 server might be causing performance problems. Could you try using iperf3.9 as the server and iperf3.16 as the client, then run the same tests to compare the results?

[jasonleschnik]

I am reporting that I also hit the same issue. I had a use case where UDP loss was suspected across a private WAN circuit, Unfortunately, this defect confirmed the suspicion and led to further testing/troubleshooting that ended up being incorrect based on the iperf reporting of loss. I've since changed to nuttcp for high-speed UDP testing.

[davidBar-On]

@IlorDash, I hope you are still interested in a solution to this issue or at least are willing to test possible solution.

I now suspect that the issue is the select() (with a timeout) that is done before reading input, to make sure that input is available, so the read will not be stuck forever. Since each stream is handled by a separate thread (since 3.16), I believe that this is done to make sure that the main thread will not get stuck when waiting for control messages, etc. Therefore, I created a version that don't do the select when reading from data streams.

Will you be able to test using this iperf3 version and see if it improves the UDP message loss?

It is available in my iperf3 repository: https://github.com/davidBar-On/iperf, branch "issue-1707-UDP-higher-packet-loss-compared-to-nuttcp". (To get it, use git clone -b issue-1707-UDP-higher-packet-loss-compared-to-nuttcp https://github.com/davidBar-On/iperf.)

[IlorDash]

Hi @davidBar-On,

I’ve tested iperf3 with your version and can confirm that the issue with excessive UDP drops has been resolved! Below are the tests results:

# iperf3 -v
iperf 3.17.1+ (cJSON 1.7.15)
Linux hostname 5.10.179 #1 SMP Wed Oct 16 11:12:07 MSK 2024 aarch64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind s

# iperf3 -4s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.15.1, port 34478
[  5] local 192.168.15.158 port 5201 connected to 192.168.15.1 port 38502
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  11.9 MBytes   100 Mbits/sec  0.045 ms  0/8642 (0%)  
...
[  5] 119.00-120.00 sec  11.9 MBytes   100 Mbits/sec  0.044 ms  0/8633 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-120.00 sec  1.40 GBytes   100 Mbits/sec  0.044 ms  0/1035904 (0%)  receiver

$ iperf3 -c 192.168.15.158 -u -b0 -t 120
Connecting to host 192.168.15.158, port 5201
[  6] local 192.168.15.1 port 47699 connected to 192.168.15.158 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  6]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  82047  
...
[  6] 119.00-120.00 sec   113 MBytes   949 Mbits/sec  81963  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  6]   0.00-120.00 sec  13.3 GBytes   949 Mbits/sec  0.000 ms  0/9825698 (0%)  sender
[  6]   0.00-120.00 sec  13.2 GBytes   943 Mbits/sec  0.015 ms  54422/9825693 (0.55%)  receiver

And overhead measurements:

Samples: 442K of event 'cpu-clock', Event count (approx.): 110634500000         
  Overhead  Command  Shared Object          Symbol                              
+   14.05%  iperf3   libc.so.6              [.] read                            
+   13.34%  iperf3   [kernel.kallsyms]      [k] do_csum                         
+   12.12%  iperf3   [kernel.kallsyms]      [k] do_el0_svc                      
+   11.65%  iperf3   [kernel.kallsyms]      [k] __local_bh_enable_ip            
+    7.24%  iperf3   [kernel.kallsyms]      [k] arch_local_irq_restore                                  

Can I expect your changes to be included in an upcoming release of iperf3?

[davidBar-On]

Thanks for testing and I am glad it helps with the issue. Regarding whether it will be included in iperf3 release, this is up to the iperf3 maintenance team. I will submit a PR with the change later, and if I didn't miss something about the reason for the select(), I hope it will be merged into the mainline.

[davidBar-On]

Submitted PR #1787 with the suggested change.