roachtest: ycsb/A/nodes=3/cpu=32 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/e1b3d0fdf11783203e76e7a2b3add59e8562a58d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=ycsb/A/nodes=3/cpu=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1249112&tab=buildLog

The test failed on master:
	cluster.go:1107,ycsb.go:27,ycsb.go:65,test.go:1237: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1249112-ycsb-a-nodes-3-cpu-32:1-3 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned:
		stderr:
		
		stdout:
		teamcity-1249112-ycsb-a-nodes-3-cpu-32: putting (dist) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach
		.................................................................................................................................
		   1: done
		   2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 [email protected]:./cockroach
		Warning: Permanently added '34.73.228.172' (ECDSA) to the list of known hosts.
		Connection to 34.73.228.172 closed by remote host.
		lost connection
		: exit status 1
		   3: done
		I190419 17:46:55.385055 1 cluster_synced.go:965  put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 failed
		: exit status 1

[ajwerner]

This failed to set up the cluster with

Connection to 34.73.228.172 closed by remote host.
lost connection

In roachprod put. I'd love to understand why this happened but this run isn't going to tell us. Maybe we should retry this and maybe we should set some flags like -vvv on the retry

[tbg]

Similar mysteries exist with roachprod start: #36963 (n12 there is not dead).

[ajwerner]

Alright, I'll try creating a roachprod with -vvv logging on stderr to the various ssh commands and try to repro

[tbg]

cc #37001