Merge #37001

37001: roachprod: increase concurrent unauthenticated SSH connections r=nvanbenschoten a=nvanbenschoten This PR bumps the permitted number of concurrent unauthenticated SSH connections from 10 to 64. Above this limit, sshd starts randomly dropping connections. It's possible this is what we have been running into with the frequent "Connection closed by remote host" errors in roachtests. See: - https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Load_Balancing - http://edoceo.com/notabene/ssh-exchange-identification Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]>
cockroachdb · Apr 24, 2019 · d386a15 · d386a15
2 parents e1864e2 + 24cca0f
commit d386a15
Show file tree

Hide file tree

Showing 2 changed files with 14 additions and 0 deletions.
diff --git a/pkg/cmd/roachprod/vm/aws/support.go b/pkg/cmd/roachprod/vm/aws/support.go
@@ -87,6 +87,14 @@ echo -e "\nmakestep 0.1 3" | sudo tee -a /etc/chrony/chrony.conf
 sudo /etc/init.d/chrony restart
 sudo chronyc -a waitsync 30 0.01 | sudo tee -a /root/chrony.log
 
+# sshguard can prevent frequent ssh connections to the same host. Disable it.
+sudo service sshguard stop
+# increase the number of concurrent unauthenticated connections to the sshd
+# daemon. See https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Load_Balancing.
+# By default, only 10 unauthenticated connections are permitted before sshd
+# starts randomly dropping connections.
+sudo sh -c 'echo "MaxStartups 64:30:128" >> /etc/ssh/sshd_config'
+sudo service sshd restart
 # increase the default maximum number of open file descriptors for
 # root and non-root users. Load generators running a lot of concurrent
 # workers bump into this often.

diff --git a/pkg/cmd/roachprod/vm/gce/utils.go b/pkg/cmd/roachprod/vm/gce/utils.go
@@ -77,6 +77,12 @@ fi
 sudo chmod 777 /mnt/data1
 # sshguard can prevent frequent ssh connections to the same host. Disable it.
 sudo service sshguard stop
+# increase the number of concurrent unauthenticated connections to the sshd
+# daemon. See https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Load_Balancing.
+# By default, only 10 unauthenticated connections are permitted before sshd
+# starts randomly dropping connections.
+sudo sh -c 'echo "MaxStartups 64:30:128" >> /etc/ssh/sshd_config'
+sudo service sshd restart
 # increase the default maximum number of open file descriptors for
 # root and non-root users. Load generators running a lot of concurrent
 # workers bump into this often.