roachtest: version/mixedWith=2.1.6/nodes=5 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/bf399d2677783dc1eea7f5ede6d4561f95c0ea10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=version/mixedWith=2.1.6/nodes=5 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1234662&tab=buildLog

The test failed on master:
	cluster.go:1295,version.go:140,version.go:205,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod stop teamcity-1234662-version-mixedwith-2-1-6-nodes-5:5 returned:
		stderr:
		
		stdout:
		teamcity-1234662-version-mixedwith-2-1-6-nodes-5: stopping and waiting........................................................................................................................
		0: exit status 255: Connection to 104.196.7.68 closed by remote host.
		
		I190411 21:05:43.953125 1 cluster_synced.go:1499  command failed
		: exit status 1
	cluster.go:1688,version.go:225,version.go:240,test.go:1237: Goexit() was called
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,version.go:225,version.go:240,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1234662-version-mixedwith-2-1-6-nodes-5 --oneshot --ignore-empty-nodes: exit status 1 6: skipped
		3: 5356
		2: 5293
		5: dead
		1: 5564
		4: 5273
		Error:  5: dead

[tbg]

n5 is indeed down, but it was shut down gracefully. I think this is one of these cases where roachprod start hung until things tore themselves down. It's hard to see from the logs, but note that the test ran for 2:30h which it's not supposed to

2h34m35s    49992            1.0            2.3     23.1     23.1     23.1     23.1 stockLevel

[tbg]

(the real problem here is that the logging isn't good enough)

[tbg]

#37001