Slow Bitrate Numbers After Upgrading 3.1.3 to 3.10.1

[ryanburnett]

Context

First off, thank you very much for your hard work on iperf. It's truly amazing software.

After upgrading iperf3 to 3.10.1 my bitrate numbers dropped by around 600 Mbits/sec.

When I used iperf 3.1.3 from https://iperf.fr/iperf-download.php I would always get a bitrate at least 900 Mbits/sec.

Speedtest.net reports 936 Mbps down https://www.speedtest.net/result/11599861093

I have two MacBook Pros connected to the same switch with Cat6 and USB-C to Gigabit Ethernet adapters.

• Version of iperf3:
  3.10.1

• Hardware:
  MacBook Pro 16" 2019 (server)
  MacBook Pro 15" 2019 (client)
  Ubiquiti UniFi Dream Machine Pro
  Ubiquiti UniFi Switch Lite 16 PoE
  Cable Matters USB-C to Gigabit Ethernet Adapter
  Monoprice Cat6 Ethernet

• Operating system:
  macOS 10.15.7

• Other relevant information:
  Installed iperf3 3.10.1 via Homebrew https://formulae.brew.sh/formula/iperf3

Bug Report

• Expected Behavior:
  Bitrate at least 900 Mbits/sec.

• Actual Behavior:
  Bitrate around 327 Mbits/sec.

MacBook Pro 16" 2019
iperf3 -s

Server listening on 5201 (test #1)
Accepted connection from 10.1.1.70, port 50112
[  5] local 10.1.1.79 port 5201 connected to 10.1.1.70 port 50113
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  44.1 MBytes   370 Mbits/sec                  
[  5]   1.00-2.00   sec  38.2 MBytes   321 Mbits/sec                  
[  5]   2.00-3.00   sec  37.7 MBytes   316 Mbits/sec                  
[  5]   3.00-4.00   sec  38.7 MBytes   324 Mbits/sec                  
[  5]   4.00-5.00   sec  37.9 MBytes   318 Mbits/sec                  
[  5]   5.00-6.00   sec  36.2 MBytes   304 Mbits/sec                  
[  5]   6.00-7.00   sec  40.4 MBytes   339 Mbits/sec                  
[  5]   7.00-8.00   sec  36.8 MBytes   309 Mbits/sec                  
[  5]   8.00-9.00   sec  38.0 MBytes   319 Mbits/sec                  
[  5]   9.00-10.00  sec  41.8 MBytes   350 Mbits/sec                  
[  5]  10.00-10.01  sec   194 KBytes   134 Mbits/sec                  
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   390 MBytes   327 Mbits/sec                  receiver

MacBook Pro 15" 2019
iperf3 -c 10.1.1.79

Connecting to host 10.1.1.79, port 5201
[  5] local 10.1.1.70 port 50113 connected to 10.1.1.79 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  45.5 MBytes   381 Mbits/sec                  
[  5]   1.00-2.00   sec  37.5 MBytes   315 Mbits/sec                  
[  5]   2.00-3.00   sec  38.2 MBytes   320 Mbits/sec                  
[  5]   3.00-4.00   sec  38.9 MBytes   326 Mbits/sec                  
[  5]   4.00-5.00   sec  37.7 MBytes   317 Mbits/sec                  
[  5]   5.00-6.00   sec  35.9 MBytes   301 Mbits/sec                  
[  5]   6.00-7.00   sec  40.5 MBytes   339 Mbits/sec                  
[  5]   7.00-8.00   sec  36.9 MBytes   310 Mbits/sec                  
[  5]   8.00-9.00   sec  38.5 MBytes   322 Mbits/sec                  
[  5]   9.00-10.00  sec  41.0 MBytes   345 Mbits/sec                  
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   391 MBytes   328 Mbits/sec                  sender
[  5]   0.00-10.01  sec   390 MBytes   327 Mbits/sec                  receiver

iperf Done.

• Steps to Reproduce:
  Install iperf3 3.10.1 via Homebrew https://formulae.brew.sh/formula/iperf3
  brew install iperf3
  MacBook Pro 16" 2019
  iperf3 -s
  MacBook Pro 15" 2019
  iperf3 -c 10.1.1.79

• Possible Solution:
  Uninstalling iperf 3.10.1 and reinstalling 3.1.3 (not ideal).

I would appreciate any insight you have on this. Thank you for your time.

[davidBar-On]

@ryanburnett, it may help if you can provide more information:

1. For both the old and new versions, run the client with -V (verbose) option. The most important is the MSS and "byte blocks" size set in the two cases (there is no need to share each interval statistics).
2. Run both old and new version using reduced message size, e.g. -l 1300. If the issue is related to fragmentation/segmentation of the default 128KB message length for TCP, then there should not be a difference between the versions is this case.
3. Run 3.10.1 using UDP with 600Mbps - -u -b 600M. If the issue is related to TCP, this test should work o.k.

[ryanburnett]

@davidBar-On thanks for your reply.

Here are the server and client results from 3.10.1 and 3.1.3. The numbers are consistent now, I'm not sure what's wrong.

I also ran another speedtest and am still getting 936 Mbps down https://www.speedtest.net/result/11622602284

3.10.1

MacBook Pro 16" 2019
iperf3 -s

iperf 3.10.1
Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~ 1/RELEASE_X86_64 x86_64
Server listening on 5201 (test #1)
Time: Wed, 23 Jun 2021 16:42:50 UTC
Accepted connection from 10.1.1.70, port 61596
      Cookie: mx3bnuvggu37oltvw7pxmbsnmcfryxenfvx6
      Target Bitrate: 600000000
[  5] local 10.1.1.218 port 5201 connected to 10.1.1.70 port 56804
Starting Test: protocol: UDP, 1 streams, 1300 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  71.1 MBytes   596 Mbits/sec  0.025 ms  0/57350 (0%)  
[  5]   1.00-2.00   sec  30.9 MBytes   258 Mbits/sec  0.016 ms  27153/52052 (52%)  
[  5]   2.00-3.00   sec  21.7 MBytes   183 Mbits/sec  0.007 ms  38721/56258 (69%)  
[  5]   3.00-4.00   sec  25.8 MBytes   216 Mbits/sec  0.005 ms  41477/62250 (67%)  
[  5]   4.00-5.01   sec  23.7 MBytes   198 Mbits/sec  0.011 ms  33021/52160 (63%)  
[  5]   5.01-6.00   sec  52.3 MBytes   441 Mbits/sec  0.075 ms  22669/64848 (35%)  
[  5]   6.00-7.00   sec  72.6 MBytes   609 Mbits/sec  0.015 ms  0/58593 (0%)  
[  5]   7.00-8.00   sec  71.5 MBytes   600 Mbits/sec  0.018 ms  0/57660 (0%)  
[  5]   8.00-9.00   sec  71.6 MBytes   600 Mbits/sec  0.044 ms  0/57736 (0%)  
[  5]   9.00-10.00  sec  71.4 MBytes   599 Mbits/sec  0.038 ms  0/57606 (0%)  
[  5]  10.00-10.01  sec   443 KBytes   660 Mbits/sec  0.005 ms  0/349 (0%)  
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5] (sender statistics not available)
[  5]   0.00-10.01  sec   513 MBytes   430 Mbits/sec  0.005 ms  163041/576862 (28%)  receiver

MacBook Pro 15" 2019
iperf3 -c 10.1.1.218 -V -l 1300 -u -b 600M

iperf 3.10.1
Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64 x86_64
Control connection MSS 1448
Time: Wed, 23 Jun 2021 16:42:50 UTC
Connecting to host 10.1.1.218, port 5201
      Cookie: mx3bnuvggu37oltvw7pxmbsnmcfryxenfvx6
      Target Bitrate: 600000000
[  5] local 10.1.1.70 port 56804 connected to 10.1.1.218 port 5201
Starting Test: protocol: UDP, 1 streams, 1300 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  71.5 MBytes   600 Mbits/sec  57668  
[  5]   1.00-2.00   sec  69.3 MBytes   581 Mbits/sec  55872  
[  5]   2.00-3.00   sec  73.8 MBytes   619 Mbits/sec  59492  
[  5]   3.00-4.00   sec  71.5 MBytes   600 Mbits/sec  57704  
[  5]   4.00-5.00   sec  71.5 MBytes   600 Mbits/sec  57703  
[  5]   5.00-6.00   sec  71.5 MBytes   600 Mbits/sec  57683  
[  5]   6.00-7.00   sec  71.5 MBytes   600 Mbits/sec  57700  
[  5]   7.00-8.00   sec  71.6 MBytes   601 Mbits/sec  57739  
[  5]   8.00-9.00   sec  71.5 MBytes   599 Mbits/sec  57654  
[  5]   9.00-10.00  sec  71.5 MBytes   600 Mbits/sec  57693  
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   715 MBytes   600 Mbits/sec  0.000 ms  0/576908 (0%)  sender
[  5]   0.00-10.01  sec   513 MBytes   430 Mbits/sec  0.005 ms  163041/576862 (28%)  receiver
CPU Utilization: local/sender 49.2% (9.4%u/39.8%s), remote/receiver 25.0% (5.4%u/19.6%s)

iperf Done.

3.1.3

MacBook Pro 16" 2019
iperf3 -s

Server listening on 5201
Accepted connection from 10.1.1.70, port 61936
[  5] local 10.1.1.218 port 5201 connected to 10.1.1.70 port 61937
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  38.0 MBytes   319 Mbits/sec                  
[  5]   1.00-2.00   sec  44.3 MBytes   371 Mbits/sec                  
[  5]   2.00-3.00   sec  39.5 MBytes   331 Mbits/sec                  
[  5]   3.00-4.00   sec  39.4 MBytes   331 Mbits/sec                  
[  5]   4.00-5.00   sec  44.5 MBytes   373 Mbits/sec                  
[  5]   5.00-6.00   sec  46.8 MBytes   392 Mbits/sec                  
[  5]   6.00-7.00   sec  45.1 MBytes   379 Mbits/sec                  
[  5]   7.00-8.00   sec  53.9 MBytes   452 Mbits/sec                  
[  5]   8.00-9.00   sec  38.4 MBytes   322 Mbits/sec                  
[  5]   9.00-10.00  sec  44.1 MBytes   368 Mbits/sec                  
[  5]  10.00-10.10  sec  0.00 Bytes  0.00 bits/sec                 
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.10  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.10  sec   434 MBytes   360 Mbits/sec                  receiver

MacBook Pro 15" 2019
iperf3 -c 10.1.1.218 -V -l 1300

iperf 3.1.3
Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64 x86_64
Time: Wed, 23 Jun 2021 17:59:59 GMT
Connecting to host 10.1.1.218, port 5201
      Cookie: MacBook-Pro.local.1624471183.96
      TCP MSS: 1448 (default)
[  4] local 10.1.1.70 port 61937 connected to 10.1.1.218 port 5201
Starting Test: protocol: TCP, 1 streams, 1300 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  39.0 MBytes   327 Mbits/sec                  
[  4]   1.00-2.00   sec  44.1 MBytes   370 Mbits/sec                  
[  4]   2.00-3.00   sec  39.4 MBytes   331 Mbits/sec                  
[  4]   3.00-4.00   sec  39.4 MBytes   330 Mbits/sec                  
[  4]   4.00-5.00   sec  44.8 MBytes   376 Mbits/sec                  
[  4]   5.00-6.00   sec  46.9 MBytes   394 Mbits/sec                  
[  4]   6.00-7.00   sec  44.9 MBytes   376 Mbits/sec                  
[  4]   7.00-8.00   sec  54.1 MBytes   454 Mbits/sec                  
[  4]   8.00-9.00   sec  38.2 MBytes   320 Mbits/sec                  
[  4]   9.00-10.01  sec  44.1 MBytes   369 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   435 MBytes   365 Mbits/sec                  sender
[  4]   0.00-10.01  sec   434 MBytes   364 Mbits/sec                  receiver
CPU Utilization: local/sender 27.5% (0.9%u/26.7%s), remote/receiver 4.2% (0.6%u/3.5%s)

iperf Done.

Any idea why my iPerf3 results are so low?

Thanks again!

[davidBar-On]

It seems that the issue is related to the MSS. Since it is only 1448 bytes, the default 128KB TCP message is segmented into many about 1400 bytes message and this is causing the reduced throughput (as can be seen with the -l 1300. Since UDP does not use a transmission control mechanism, it is able to get the 600Mbps with 3.10.1 (and probably can get to about the 900Mbps).

What I don't understand is how the 3.1.3 is able to get the 900Mbps. Can you run again the original TCP tests, but now with the -V verbose flag (but WITHOUT the -l), so we can see the MSS and "byte blocks" for both versions?

[ryanburnett]

Here's the test with -V on 3.10.1 - I hit 941 Mbits/sec receiver bitrate!

MacBook Pro 16" 2019
iperf3 -s

iperf 3.10.1
Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64 x86_64
Server listening on 5201 (test #1)
Accepted connection from 10.1.1.97, port 63775
[  5] local 10.1.1.218 port 5201 connected to 10.1.1.97 port 63776
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  
[  5]  10.00-10.01  sec  1.65 MBytes   938 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.10 GBytes   941 Mbits/sec                  receiver

MacBook Pro 15" 2019
iperf3 -c 10.1.1.218 -V

iperf 3.10.1
Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May  6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64 x86_64
Control connection MSS 1448
Time: Thu, 24 Jun 2021 01:41:58 UTC
Connecting to host 10.1.1.218, port 5201
      Cookie: 4fcwq23aw46dhq5hvwntxnaoddtwbq5nnsar
      TCP MSS: 1448 (default)
[  5] local 10.1.1.97 port 63776 connected to 10.1.1.218 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   114 MBytes   955 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec                  sender
[  5]   0.00-10.01  sec  1.10 GBytes   941 Mbits/sec                  receiver
CPU Utilization: local/sender 39.8% (2.1%u/37.7%s), remote/receiver 23.6% (3.8%u/19.8%s)

iperf Done.

Did -V do the trick? Or is there something else I'm missing?

Thanks again.

[davidBar-On]

Did -V do the trick? Or is there something else I'm missing?

The -V should just add more information to the output and should not have any effect on the test itself. Can it be that there was a temporary network problem when you first run the test? In any case, I suggest that you will try the test without the -V to make sure it really doesn't affect the test results.

[ryanburnett]

@davidBar-On Thank you so much for your help.

I recently moved my Philips Hue Bridge to a different port on this switch. I forgot to change the old port's configuration from my IoT VLAN back to the LAN. The IoT VLAN port was the one I originally used for the client Mac for this test. Since the client Mac was connected to a different network, the Mac was failing over to WiFi, which is why the iPerf3 numbers were so low.

I turned off WiFi on the client Mac, connected it to the IoT port, ran the -c test again, and got:

iperf3: error - unable to connect to server: Operation timed out

When I set the port back to the LAN, I get a successful result.

MacBook Pro 16" 2019
iperf3 -s

Server listening on 5201 (test #1)
Accepted connection from 10.1.1.97, port 50551
[  5] local 10.1.1.218 port 5201 connected to 10.1.1.97 port 50552
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   938 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  
[  5]  10.00-10.01  sec  1.41 MBytes   918 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.10 GBytes   941 Mbits/sec                  receiver

MacBook Pro 15" 2019
iperf3 -c 10.1.1.218

Connecting to host 10.1.1.218, port 5201
[  5] local 10.1.1.97 port 50552 connected to 10.1.1.218 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   114 MBytes   954 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  sender
[  5]   0.00-10.01  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.

Thanks again for pinpointing this problem. I really appreciate your help.

[davidBar-On]

Thanks for the detailed information about the problem. Can help with evaluation of future issues with iperf3.