Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression in 3.1/3.2: iperf server fails after ~15 seconds with "Bad file descriptor" #645

Closed
fionn opened this issue Oct 3, 2017 · 5 comments · Fixed by #648
Closed

Regression in 3.1/3.2: iperf server fails after ~15 seconds with "Bad file descriptor" #645

fionn opened this issue Oct 3, 2017 · 5 comments · Fixed by #648
Assignees
@bmah888
Labels
bug

Comments

@fionn
Copy link

fionn commented Oct 3, 2017

  • Version of iperf3: 3.2, 3.2rc1 and latest commit on master (cd5d89d).

  • Hardware: MBP (Arch Linux, 2015), MBP (MacOS, 2017), VPS (Arch Linux).

  • Operating system (and distribution, if any): Arch Linux, MacOS.

Bug Report

  • Expected behaviour: setting -t 0, I expect iperf to run indefinitely.

  • Actual behaviour: the connection is dropped after about 15 seconds.

  • Steps to reproduce: run iperf 3.2 as a server with iperf3 -s and a client (any version) with iperf3 -c $HOST -t 0 (can be entirely local, with HOST=localhost); after ~15 seconds, the client will report

iperf3: error - control socket has closed unexpectedly

and the server will report

iperf3: error - select failed: Bad file descriptor

(this can also be reproduced in reverse mode).
@bmah888
Copy link
Contributor

bmah888 commented Oct 3, 2017

This is a weird problem on a couple levels. I see it on the tip of the master and 3.1-STABLE branches, but not the tip of 3.0-STABLE (tested variously on macOS Sierra and CentOS 7). The test shouldn't stop like that, so there's definitely a bug there.

I'm not sure what the correct behavior is, on the other hand. Your "run indefinitely" makes some sense, but on the other hand it'd also make sense to not allow this case at all, so that a client can't conduct a DoS attack against a server (a counter-argument is that a server should be able to specify a maximum test length).

I'm going to dig a little more into this and try to understand what's going on. So far I have the following interesting result:

Server (iperf-master), client (iperf-master): Failure
Server (iperf-3.0-stable), client (iperf-master): OK
Server (iperf-master), client (iperf-3.0-stable): Failure
Server (iperf-3.0-stable), client (iperf-3.0-stable): OK

This so far implies that there's a problem on the server side with newer iperf.

BTW thanks for a well-formulated bug report!

@bmah888 bmah888 self-assigned this Oct 3, 2017
@bmah888 bmah888 added the bug label Oct 3, 2017
@bmah888 bmah888 changed the title Regression in 3.2: iperf server fails after ~15 seconds with "Bad file descriptor" Regression in 3.1/3.2: iperf server fails after ~15 seconds with "Bad file descriptor" Oct 3, 2017
@bmah888
Copy link
Contributor

bmah888 commented Oct 3, 2017

What's happening here is that if -t 0 is set on the client, that value isn't passed to the server (which then assumes the default duration of 10 seconds). The server sets a sort of watchdog timer for 5 seconds past the total test length (which is the value of -t and -i), so that's where the 15 seconds comes from. The solution seems to be to pass the value of -t to the server unconditionally, but I'm digging around to see if this makes some other problem.

The behavior for setting a timer on the server side was added in PR #446, although it didn't really create the bug we're seeing here.

bmah888 added a commit that referenced this issue Oct 5, 2017
@bmah888
Allow the client to pass a duration of 0 to the server.
2e51e48 
Seems to fix #645.
@bmah888 bmah888 mentioned this issue Oct 5, 2017
Merged
@fionn
Copy link
Author

fionn commented Oct 11, 2017

I just tried issue-645 (2e51e48); it obviously works perfectly. Thanks for explaining why this happens.

I don't see this as a DoS threat, since clients can send -t 86400, or repeatedly make connections. As you said, it seems like it should be up to the server to allow or disallow these connections. I suspect that with authentication this matters even less.

bmah888 added a commit that referenced this issue Oct 11, 2017
@bmah888
Allow the client to pass a duration of 0 to the server. (#648)
720a0a3 
Seems to fix #645.
@bmah888
Copy link
Contributor

bmah888 commented Oct 11, 2017

@iamfionn: Thanks for the testing!

@bmah888 bmah888 mentioned this issue Oct 17, 2017
Closed
@tsrofj
Copy link

tsrofj commented Dec 12, 2017 •
edited
Loading

I have this same/similar issue:
Server side is Amazon AWS Linux AMI (198-51-100-112), running build from esnet repo, and client is an ARM v7l (203.0.113.47) with iperf3 compiled from esnet source tarball this morning

ec2-user@ip-198-51-100-112 19:00:25 UTC ~ $ cat /etc/*release
NAME="Amazon Linux AMI"
VERSION="2017.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.09"
PRETTY_NAME="Amazon Linux AMI 2017.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2017.09

ec2-user@ip-198-51-100-112 19:00:34 UTC ~ $  iperf3 -v
iperf 3.3 (cJSON 1.5.2)
Linux ip-198-51-100-112 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, authentication

ec2-user@ip-198-51-100-112 19:03:07 UTC ~ $ iperf3 -s -f m -i 1 -V
iperf 3.3
Linux ip-198-51-100-112 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Tue, 12 Dec 2017 19:05:43 GMT
Accepted connection from 203.0.113.47, port 25473
      Cookie: fhj3mal6uf5bcnggszkq5rkpp5iskjj6z65h
      TCP MSS: 0 (default)
[  5] local 198.51.100.112 port 5201 connected to 203.0.113.47 port 14525
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.6 KBytes  0.10 Mbits/sec                  
[  5]   1.00-2.00   sec  32.5 KBytes  0.27 Mbits/sec                  
[  5]   2.00-3.00   sec  37.1 KBytes  0.30 Mbits/sec                  
[  5]   3.00-4.00   sec  39.4 KBytes  0.32 Mbits/sec                  
[  5]   4.00-5.00   sec  42.9 KBytes  0.35 Mbits/sec                  
[  5]   5.00-6.00   sec  30.2 KBytes  0.25 Mbits/sec                  
[  5]   6.00-7.00   sec  59.2 KBytes  0.48 Mbits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 Mbits/sec                  
[  5]   8.00-9.00   sec   309 KBytes  2.53 Mbits/sec                  
[  5]   9.00-10.00  sec   138 KBytes  1.13 Mbits/sec                  
[  5]  10.00-10.86  sec   132 KBytes  1.26 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5] (sender statistics not available)
[  5]   0.00-10.86  sec   832 KBytes  0.63 Mbits/sec                  receiver
CPU Utilization: local/receiver 0.0% (0.0%u/0.0%s), remote/sender 0.0% (0.0%u/0.0%s)
rcv_tcp_congestion cubic
iperf3: the client has unexpectedly closed the connection
iperf 3.3
Linux ip-198-51-100-112 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Tue, 12 Dec 2017 19:06:22 GMT
Accepted connection from 203.0.113.47, port 10596
      Cookie: r3ktyinenamgwlb47ndk6vtvqme6ks2pc3cy
      TCP MSS: 0 (default)
[  5] local 198.51.100.112 port 5201 connected to 203.0.113.47 port 6223
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 20 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.6 KBytes  0.10 Mbits/sec                  
[  5]   1.00-2.00   sec  47.6 KBytes  0.39 Mbits/sec                  
[  5]   2.00-3.00   sec  38.3 KBytes  0.31 Mbits/sec                  
[  5]   3.00-4.00   sec  49.9 KBytes  0.41 Mbits/sec                  
[  5]   4.00-5.00   sec  34.8 KBytes  0.29 Mbits/sec                  
[  5]   5.00-6.00   sec  44.1 KBytes  0.36 Mbits/sec                  
[  5]   6.00-7.00   sec  38.3 KBytes  0.31 Mbits/sec                  
[  5]   7.00-8.00   sec  44.1 KBytes  0.36 Mbits/sec                  
[  5]   8.00-9.00   sec  36.0 KBytes  0.29 Mbits/sec                  
[  5]   9.00-10.00  sec  45.2 KBytes  0.37 Mbits/sec                  
[  5]  10.00-11.00  sec  37.1 KBytes  0.30 Mbits/sec                  
[  5]  11.00-12.00  sec  41.8 KBytes  0.34 Mbits/sec                  
[  5]  12.00-13.00  sec  45.2 KBytes  0.37 Mbits/sec                  
[  5]  13.00-14.00  sec  41.8 KBytes  0.34 Mbits/sec                  
[  5]  14.00-15.00  sec  41.8 KBytes  0.34 Mbits/sec                  
[  5]  15.00-16.00  sec  46.4 KBytes  0.38 Mbits/sec                  
[  5]  16.00-17.00  sec  47.6 KBytes  0.39 Mbits/sec                  
[  5]  17.00-18.00  sec  37.1 KBytes  0.30 Mbits/sec                  
[  5]  18.00-19.00  sec  47.6 KBytes  0.39 Mbits/sec                  
[  5]  19.00-20.00  sec  40.6 KBytes  0.33 Mbits/sec                  
[  5]  20.00-21.00  sec  44.1 KBytes  0.36 Mbits/sec                  
[  5]  21.00-22.00  sec  46.4 KBytes  0.38 Mbits/sec                  
[  5]  22.00-23.00  sec  39.4 KBytes  0.32 Mbits/sec                  
[  5]  23.00-24.00  sec  47.6 KBytes  0.39 Mbits/sec                  
iperf3: error - select failed: Bad file descriptor
iperf 3.3
Linux ip-198-51-100-112 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
ec2-user@ip-198-51-100-112 19:12:54 UTC ~ $ cd /etc/yum.repos.d/
ec2-user@ip-198-51-100-112 19:13:01 UTC /etc/yum.repos.d $ ll
total 36
-rw-r--r-- 1 root root 1014 Nov  1 19:45 amzn-main.repo
-rw-r--r-- 1 root root  327 Nov  1 19:45 amzn-nosrc.repo
-rw-r--r-- 1 root root 1041 Nov  1 19:45 amzn-preview.repo
-rw-r--r-- 1 root root 1041 Nov  1 19:45 amzn-updates.repo
-rw-r--r-- 1 root root  951 Oct  2 17:44 epel.repo
-rw-r--r-- 1 root root 1050 Oct  2 17:44 epel-testing.repo
-rw-r--r-- 1 root root  316 Jun 26 20:47 perfSONAR.repo
-rw-r--r-- 1 root root 1688 Jun 26 20:47 perfSONAR-Vault.repo
-rw-r--r-- 1 root root  364 Jun 26 20:47 perfSONAR-web100_kernel.repo
ec2-user@ip-198-51-100-112 19:13:02 UTC /etc/yum.repos.d $ 



[default@gw /]$ cat /etc/*release
NAME=Buildroot
VERSION=2016.02-dirty
ID=buildroot
VERSION_ID=2016.02
PRETTY_NAME="Buildroot 2016.02"

[default@gw /]$ iperf3 -v
iperf 3.3 (cJSON 1.5.2)
Linux gw 3.14.0501010-000145R20-2.5.4.611-xilinx #1 SMP PREEMPT Fri Jul 28 01:12:18 CDT 2017 armv7l
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing

[default@gw /]$ iperf3 -c 54.80.173.204 -f m -i 1 -V
iperf 3.3
Linux gw 3.14.0501010-000145R20-2.5.4.611-xilinx #1 SMP PREEMPT Fri Jul 28 01:12:18 CDT 2017 armv7l
Control connection MSS 1188
Time: Tue, 12 Dec 2017 19:05:44 GMT
Connecting to host 198.51.100.112, port 5201
      Cookie: fhj3mal6uf5bcnggszkq5rkpp5iskjj6z65h
      TCP MSS: 1188 (default)
[  5] local 203.0.113.47 port 60331 connected to 54.80.173.204 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.1 KBytes  0.36 Mbits/sec    0   23.2 KBytes       
[  5]   1.00-2.00   sec  76.6 KBytes  0.63 Mbits/sec    0   55.7 KBytes       
[  5]   2.00-3.00   sec  69.6 KBytes  0.57 Mbits/sec    0   91.7 KBytes       
[  5]   3.00-4.00   sec   104 KBytes  0.86 Mbits/sec    0    132 KBytes       
[  5]   4.00-5.00   sec   157 KBytes  1.28 Mbits/sec    0    175 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec    0    205 KBytes       
[  5]   6.00-7.00   sec   233 KBytes  1.91 Mbits/sec    1    254 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 Mbits/sec    0    182 KBytes       
[  5]   8.00-9.00   sec   300 KBytes  2.46 Mbits/sec    0    133 KBytes       
[  5]   9.00-10.00  sec   158 KBytes  1.29 Mbits/sec    0    133 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.12 MBytes  0.94 Mbits/sec    1             sender
[  5]   0.00-10.86  sec   832 KBytes  0.63 Mbits/sec                  receiver
CPU Utilization: local/sender 5.9% (0.8%u/5.1%s), remote/receiver 0.0% (0.0%u/0.0%s)
snd_tcp_congestion veno
rcv_tcp_congestion cubic

iperf Done.
[default@gw /]$ iperf3 -c 198.51.100.112 -f m -i 1 -V -t 20
iperf 3.3
Linux gw 3.14.0501010-000145R20-2.5.4.611-xilinx #1 SMP PREEMPT Fri Jul 28 01:12:18 CDT 2017 armv7l
Control connection MSS 1188
Time: Tue, 12 Dec 2017 19:06:23 GMT
Connecting to host 54.80.173.204, port 5201
      Cookie: r3ktyinenamgwlb47ndk6vtvqme6ks2pc3cy
      TCP MSS: 1188 (default)
[  5] local 203.0.113.47 port 60333 connected to 198.51.100.112 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 20 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.1 KBytes  0.36 Mbits/sec    0   23.2 KBytes       
[  5]   1.00-2.00   sec   146 KBytes  1.20 Mbits/sec    0   70.8 KBytes       
[  5]   2.00-3.00   sec   104 KBytes  0.86 Mbits/sec    0    109 KBytes       
[  5]   3.00-4.00   sec   157 KBytes  1.28 Mbits/sec    0    159 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec    0    194 KBytes       
[  5]   5.00-6.00   sec   233 KBytes  1.91 Mbits/sec    0    238 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 Mbits/sec    0    276 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 Mbits/sec    0    321 KBytes       
[  5]   8.00-9.00   sec   352 KBytes  2.88 Mbits/sec    0    356 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 Mbits/sec    0    400 KBytes       
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 Mbits/sec    0    442 KBytes       
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 Mbits/sec    0    479 KBytes       
[  5]  12.00-13.00  sec   529 KBytes  4.33 Mbits/sec    0    528 KBytes       
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 Mbits/sec    0    516 KBytes       
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 Mbits/sec    0    495 KBytes       
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 Mbits/sec    0    472 KBytes       
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 Mbits/sec    0    448 KBytes       
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 Mbits/sec    0    429 KBytes       
[  5]  18.00-19.00  sec  0.00 Bytes  0.00 Mbits/sec    0    406 KBytes       
iperf3: error - control socket has closed unexpectedly
[default@gw /]$

@esqt esqt mentioned this issue Jun 15, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bmah888 bmah888

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow the client to pass a duration of 0 to the server. esnet/iperf
3 participants
@fionn @bmah888 @tsrofj