roachtest: clearrange/checks=false failed

[cockroach-teamcity]

roachtest.clearrange/checks=false failed with artifacts on release-22.1 @ ae5fec2629502404a949fa4496decf20d67b95b4:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1
	cluster.go:1953,clearrange.go:70,clearrange.go:39,test_runner.go:875: output in run_064835.887495185_n1_cockroach_workload_fixtures_import_bank: ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1987
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1951
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:70
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_064835.887495185_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I220402 06:48:36.668668 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: pq: pausing due to error; use RESUME JOB to try to proceed once the issue is resolved, or CANCEL JOB to rollback: store 9 has insufficient remaining capacity to ingest data (remaining: 18 GiB / 5.0%, min required: 5.0%)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: clearrange/checks=false failed #78408 roachtest: clearrange/checks=false failed [A-bulkio C-test-failure GA-blocker O-roachtest O-robot T-bulkio T-storage branch-master]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-14668

[jbowens]

Dupe of #78408 (corresponding master branch issue); keeping open until it's fixed

[nicktrav]

Posting some evidence too. This time range helps see the full picture (when booting a local cluster with the tsdump).

OOM killed n10:

[ 1150.258591] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/cockroach.service,task_memcg=/system.slice/cockroach.service,task=cockroach,pid=12994,uid=1000
[ 1150.258737] Memory cgroup out of memory: Killed process 12994 (cockroach) total-vm:20462760kB, anon-rss:13621088kB, file-rss:8092kB, shmem-rss:0kB, UID:1000 pgtables:38420kB oom_score_adj:0
[ 1151.014168] oom_reaper: reaped process 12994 (cockroach), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Memory usage:

[nicktrav]

Given this was a dupe of #78408, and that issue was closed (see here for links to related tickets), I'm going to close this one out too.

[rail]

Manually synced with Jira