roachtest: import/tpch/nodes=8 failed

[cockroach-teamcity]

roachtest.import/tpch/nodes=8 failed with artifacts on release-22.1 @ d84033d9ee0dc4f35901bcc2e2311b6288e84569:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1
	monitor.go:127,import.go:312,test_runner.go:883: monitor failure: monitor command failure: unexpected node event: 6: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:312
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 6: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: import/tpch/nodes=8 failed #90021 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-22.2]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-22588

Epic CRDB-20293

[msbutler]

Node 6 died from disk stall. Handing over to storage for further investigation. From the unredacted cockroach.log on node 6:

F221219 06:22:11.020890 88 storage/pebble.go:915 ⋮ [n6] 225  disk stall detected: pebble unable to write to ‹/mnt/data1/cockroach/COCKROACHDB_REGISTRY_000001› in 21.50 seconds

[nicktrav]

This one looks like an infra flake. Closing.

[msbutler]

in the future, what's the best way to identify infra flake disk stalls? Or should we always hand these to storage?

[nicktrav]

The log line you already posted is generally a decent indication of something wonky at the device level.

disk stall detected: pebble unable to write to ...

In the Pebble logs you should see the full picture. Especially if that stall happens across multiple files, and ultimately times out (the limit is currently 20s) and causes the process to panic.

In such cases there's not too much we can do.