Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] #79887

Closed
cockroach-teamcity opened this issue Apr 13, 2022 · 66 comments · May be fixed by #80205
Closed

roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] #79887

cockroach-teamcity opened this issue Apr 13, 2022 · 66 comments · May be fixed by #80205
Assignees
@aayushshah15
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. S-1 High impact: many users impacted, serious risk of high unavailability or data loss T-kv KV Team X-stale

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 13, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 43457fc6e2f5324823cca4156d9799453753b9cc:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:875: monitor failure: monitor task failed: pq: internal error: range r386:/Table/106/1/3{694877309469296640-713305625227247616} [(n2,s2):11, (n6,s6):15, (n5,s5):14, (n3,s3):16LEARNER, next=17, gen=109, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):11,(n6,s6):15,(n5,s5):14,(n3,s3):16LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r386:/Table/106/1/3{694877309469296640-713305625227247616} [(n2,s2):11, (n6,s6):15, (n5,s5):14, (n3,s3):16LEARNER, next=17, gen=109, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):11,(n6,s6):15,(n5,s5):14,(n3,s3):16LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-15876
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 13, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Apr 13, 2022
@tbg
Copy link
Member

tbg commented Apr 14, 2022 

  Wraps: (5) pq: internal error: range r386:/Table/106/1/3{694877309469296640-713305625227247616} [(n2,s2):11, (n6,s6):15, (n5,s5):14, (n3,s3):16LEARNER, next=17, gen=109, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):11,(n6,s6):15,(n5,s5):14,(n3,s3):16LEARNER

cockroach/pkg/kv/kvserver/replica_command.go

Lines 2930 to 2935 in c99bd0c

if repls := desc.Replicas(); len(repls.VoterFullAndNonVoterDescriptors()) != len(repls.Descriptors()) {
// The caller removed all the learners and left the joint config, so there
// shouldn't be anything but voters and non_voters.
return nil, nil, errors.AssertionFailedf(
`range %s was either in a joint configuration or had learner replicas: %v`, desc, desc.Replicas())
}

None of these were recently touched.

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 6c91da48b8da78c13e4cb8675a1a98c5ec75ea06:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:875: monitor failure: monitor task failed: pq: internal error: range r246:/Table/106/1/82{28242985925390336-46671301683341312} [(n8,s8):11, (n7,s7):7, (n1,s1):12, (n4,s4):13LEARNER, next=14, gen=83, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):11,(n7,s7):7,(n1,s1):12,(n4,s4):13LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r246:/Table/106/1/82{28242985925390336-46671301683341312} [(n8,s8):11, (n7,s7):7, (n1,s1):12, (n4,s4):13LEARNER, next=14, gen=83, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):11,(n7,s7):7,(n1,s1):12,(n4,s4):13LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@nvanbenschoten
Copy link
Member

I think this is fallout from #79405. AdminRelocateRange first calls into maybeLeaveAtomicChangeReplicasAndRemoveLearners to exit the joint configuration and remove learner replicas. In that PR, we made it so that the conditional get in execChangeReplicasTxn is more specific when removing learner replicas, and now checks during a single learner removal whether the learner was already removed, instead of that any part of the descriptor changed. In such cases, we immediately declare that the conditional check should be skipped. The issue is that this allows for races where the learner we were trying to remove is now gone but a new learner has been added. So maybeLeaveAtomicChangeReplicasAndRemoveLearners no longer guarantees that it will return with a descriptor that contains no LEARNERS.

We need to change this fast-path conditional get condition to guarantee the absence of learners. That's a little awkward because the interface of execChangeReplicasTxn accepts imperative internalReplicationChanges (e.g. remove this specific learner), not declarative policies (e.g. remove all learners if you see them). Maybe that's fine, though it will lead to some odd behavior when the caller is trying to remove multiple learners. #40268 might help us avoid this behavior by allowing the caller (maybeLeaveAtomicChangeReplicasAndRemoveLearners) to call execChangeReplicasTxn once instead of once per learner.

@nvanbenschoten
Copy link
Member

#79405 has not been backported, so this is not a release blocker. We won't be able to backport #79405 until this issue is resolved.

@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 18, 2022
@AlexTalks AlexTalks added the S-1 High impact: many users impacted, serious risk of high unavailability or data loss label Apr 19, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 1fdbb16fa206c5fd77c3aa111ec40916edfb55df:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:875: monitor failure: monitor task failed: pq: internal error: range r809:/Table/106/1/10{78056471840167936-96484787598120960} [(n1,s1):6, (n2,s2):2, (n7,s7):4, (n5,s5):7LEARNER, next=8, gen=77, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):6,(n2,s2):2,(n7,s7):4,(n5,s5):7LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r809:/Table/106/1/10{78056471840167936-96484787598120960} [(n1,s1):6, (n2,s2):2, (n7,s7):4, (n5,s5):7LEARNER, next=8, gen=77, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):6,(n2,s2):2,(n7,s7):4,(n5,s5):7LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@aayushshah15 aayushshah15 mentioned this issue Apr 19, 2022
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 0add8527e82d0ae4987ea7105778c7aa6e43bd3a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: internal error: range r996:/Table/106/1/80{07103196829970432-25531512587921408} [(n7,s7):17, (n4,s4):18, (n1,s1):19, (n5,s5):20LEARNER, next=21, gen=143, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):17,(n4,s4):18,(n1,s1):19,(n5,s5):20LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r996:/Table/106/1/80{07103196829970432-25531512587921408} [(n7,s7):17, (n4,s4):18, (n1,s1):19, (n5,s5):20LEARNER, next=21, gen=143, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):17,(n4,s4):18,(n1,s1):19,(n5,s5):20LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 8fe42c18e5e2a82494fa6a34e068f4ab801f87c4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: internal error: range r748:/Table/106/1/49{29574465252052992-48002781010003968} [(n4,s4):16, (n5,s5):15, (n8,s8):17, (n3,s3):18LEARNER, next=19, gen=97, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n4,s4):16,(n5,s5):15,(n8,s8):17,(n3,s3):18LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r748:/Table/106/1/49{29574465252052992-48002781010003968} [(n4,s4):16, (n5,s5):15, (n8,s8):17, (n3,s3):18LEARNER, next=19, gen=97, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n4,s4):16,(n5,s5):15,(n8,s8):17,(n3,s3):18LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 36b190afcffec66272bd5485567757b1a106c9e0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: internal error: range r657:/Table/106/1/87{62664142905985024-81092458663938048} [(n2,s2):6, (n1,s1):7, (n5,s5):8, (n4,s4):9LEARNER, next=10, gen=65, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):6,(n1,s1):7,(n5,s5):8,(n4,s4):9LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r657:/Table/106/1/87{62664142905985024-81092458663938048} [(n2,s2):6, (n1,s1):7, (n5,s5):8, (n4,s4):9LEARNER, next=10, gen=65, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):6,(n1,s1):7,(n5,s5):8,(n4,s4):9LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 2fdc0a62df08829dd96901fffb5fb1fbb0b0997d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 72e485705aa859ef850ebc67568bc29fe9b7cfd6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 97e72bacbeb9574f09f7475a62ef45c3e228183e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ a2e1910f51593bd2ef72e1d7c615e08f95791186:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ c9e0194b19a03d55c6be92572aad3fbafc256334:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 7a9eb906ce86e2f75db637e29d46cd6604fca7b4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 01572daaf94f80f81f843723a8b58d80fe128990:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:828,kv.go:856,test_runner.go:876: monitor failure: monitor task failed: pq: cannot remove learner while snapshot is in flight
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:828
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:856
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: cannot remove learner while snapshot is in flight
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

aayushshah15 added a commit to aayushshah15/cockroach that referenced this issue May 3, 2022
@aayushshah15
kvserver: ignore errors encountered while removing live learners
4b92564 
This commit fixes a set of failures noticed in cockroachdb#79887 where a bunch of
`EXPERIMENTAL_RELOCATE` calls failed because they tried removing learner
replicas that are in the process of receiving their initial snapshot.

Removing learners in this state is disallowed as of cockroachdb#79379. This commit fixes a
subset of those failures by making the `kv50/rangelookups` roachtest ignore
these errors.

Relates to cockroachdb#79887

Release note: None
@aayushshah15 aayushshah15 mentioned this issue May 3, 2022
Merged
craig bot pushed a commit that referenced this issue May 4, 2022
@aayushshah15
Merge #80918
91e4882 
80918: kvserver: ignore errors encountered while removing live learners r=aayushshah15 a=aayushshah15

This commit fixes a set of failures noticed in #79887 where a bunch of
`EXPERIMENTAL_RELOCATE` calls failed because they tried removing learner
replicas that are in the process of receiving their initial snapshot. The test simply
randomly issues these `RELOCATE` calls so this behavior is intentional.

Removing learners in this state is disallowed as of #79379. This commit fixes a
subset of those failures by making the `kv50/rangelookups` roachtest ignore
these errors.

Relates to #79887

Release note: None

Co-authored-by: Aayush Shah <[email protected]>
@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ b0a8d28feb8befa417933d0a2ee99511b16414a1:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:848,kv.go:876,test_runner.go:884: monitor failure: monitor task failed: pq: internal error: range r492:/Table/106/1/57{21992042843971584-40420358601922560} [(n1,s1):1, (n8,s8):7, (n3,s3):8, (n2,s2):9LEARNER, next=10, gen=43, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):1,(n8,s8):7,(n3,s3):8,(n2,s2):9LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:848
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:876
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r492:/Table/106/1/57{21992042843971584-40420358601922560} [(n1,s1):1, (n8,s8):7, (n3,s3):8, (n2,s2):9LEARNER, next=10, gen=43, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):1,(n8,s8):7,(n3,s3):8,(n2,s2):9LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ aadbaf97b4e6092ad6978a28e2735715d64d9f10:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:848,kv.go:876,test_runner.go:884: monitor failure: monitor task failed: pq: internal error: range r520:/Table/106/1/19{07330680947990528-25758996705941504} [(n3,s3):15, (n5,s5):14, (n7,s7):13, (n2,s2):16LEARNER, next=17, gen=95, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):15,(n5,s5):14,(n7,s7):13,(n2,s2):16LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:848
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:876
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r520:/Table/106/1/19{07330680947990528-25758996705941504} [(n3,s3):15, (n5,s5):14, (n7,s7):13, (n2,s2):16LEARNER, next=17, gen=95, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):15,(n5,s5):14,(n7,s7):13,(n2,s2):16LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:897: monitor failure: monitor task failed: pq: internal error: range r532:/Table/106/1/16{67762576094619648-86190891852570624} [(n6,s6):13, (n7,s7):14, (n5,s5):15, (n8,s8):16LEARNER, next=17, gen=97, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):13,(n7,s7):14,(n5,s5):15,(n8,s8):16LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r532:/Table/106/1/16{67762576094619648-86190891852570624} [(n6,s6):13, (n7,s7):14, (n5,s5):15, (n8,s8):16LEARNER, next=17, gen=97, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):13,(n7,s7):14,(n5,s5):15,(n8,s8):16LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 7a1ced4a49d28f66f381e78733bcab510e2cbafe:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r601:/Table/106/1/11{14913103356071936-33341419114022912} [(n1,s1):11, (n6,s6):12, (n8,s8):13, (n3,s3):14LEARNER, next=15, gen=79, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):11,(n6,s6):12,(n8,s8):13,(n3,s3):14LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r601:/Table/106/1/11{14913103356071936-33341419114022912} [(n1,s1):11, (n6,s6):12, (n8,s8):13, (n3,s3):14LEARNER, next=15, gen=79, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):11,(n6,s6):12,(n8,s8):13,(n3,s3):14LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r1060:/Table/106/1/5{795705305875777536-814133621633730560} [(n4,s4):14, (n7,s7):15, (n6,s6):10, (n5,s5):16LEARNER, next=17, gen=81, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n4,s4):14,(n7,s7):15,(n6,s6):10,(n5,s5):16LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r1060:/Table/106/1/5{795705305875777536-814133621633730560} [(n4,s4):14, (n7,s7):15, (n6,s6):10, (n5,s5):16LEARNER, next=17, gen=81, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n4,s4):14,(n7,s7):15,(n6,s6):10,(n5,s5):16LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ fc7ae339c3f85ad2a2b28643e47d7c20768fe237:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r336:/Table/106/1/8{20060051228846080-38488366986797056} [(n3,s3):26, (n4,s4):25, (n7,s7):21, (n2,s2):27LEARNER, next=28, gen=135, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):26,(n4,s4):25,(n7,s7):21,(n2,s2):27LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r336:/Table/106/1/8{20060051228846080-38488366986797056} [(n3,s3):26, (n4,s4):25, (n7,s7):21, (n2,s2):27LEARNER, next=28, gen=135, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):26,(n4,s4):25,(n7,s7):21,(n2,s2):27LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ b47dc374417fe4ba83570784040da5650238b30e:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r619:/Table/106/1/3{694877309469296640-713305625227247616} [(n5,s5):8, (n2,s2):5, (n6,s6):9, (n3,s3):10LEARNER, next=11, gen=81, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n5,s5):8,(n2,s2):5,(n6,s6):9,(n3,s3):10LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r619:/Table/106/1/3{694877309469296640-713305625227247616} [(n5,s5):8, (n2,s2):5, (n6,s6):9, (n3,s3):10LEARNER, next=11, gen=81, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n5,s5):8,(n2,s2):5,(n6,s6):9,(n3,s3):10LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ a611410e3babe97a14152381b89060e1bafa2e27:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ ac11a55499120ffd33016feebc47b4ed43a21007:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r791:/Table/106/1/24{41751837928587264-60180153686538240} [(n6,s6):9, (n8,s8):18, (n3,s3):17, (n2,s2):19LEARNER, next=20, gen=123, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):9,(n8,s8):18,(n3,s3):17,(n2,s2):19LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r791:/Table/106/1/24{41751837928587264-60180153686538240} [(n6,s6):9, (n8,s8):18, (n3,s3):17, (n2,s2):19LEARNER, next=20, gen=123, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):9,(n8,s8):18,(n3,s3):17,(n2,s2):19LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 78caa5c3a8ecd0d0a612798a106b626edb3db2ee:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r166:/Table/106/1/45{24151518577117184-42579834335070208} [(n7,s7):23, (n6,s6):21, (n3,s3):22, (n1,s1):24LEARNER, next=25, gen=103, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):23,(n6,s6):21,(n3,s3):22,(n1,s1):24LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r166:/Table/106/1/45{24151518577117184-42579834335070208} [(n7,s7):23, (n6,s6):21, (n3,s3):22, (n1,s1):24LEARNER, next=25, gen=103, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):23,(n6,s6):21,(n3,s3):22,(n1,s1):24LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 4e8134033890c956d39020f28a0e99b21b32649f:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r312:/Table/106/1/38{42303835532908544-60732151290859520} [(n2,s2):12, (n6,s6):13, (n7,s7):14, (n4,s4):15LEARNER, next=16, gen=91, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):12,(n6,s6):13,(n7,s7):14,(n4,s4):15LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r312:/Table/106/1/38{42303835532908544-60732151290859520} [(n2,s2):12, (n6,s6):13, (n7,s7):14, (n4,s4):15LEARNER, next=16, gen=91, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n2,s2):12,(n6,s6):13,(n7,s7):14,(n4,s4):15LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 3c9b17113488d2ee6929936aa6ec48396f3ed71c:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r663:/Table/106/1/85{78380985326469120-96809301084422144} [(n3,s3):6, (n5,s5):2, (n2,s2):7, (n4,s4):8LEARNER, next=9, gen=83, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):6,(n5,s5):2,(n2,s2):7,(n4,s4):8LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r663:/Table/106/1/85{78380985326469120-96809301084422144} [(n3,s3):6, (n5,s5):2, (n2,s2):7, (n4,s4):8LEARNER, next=9, gen=83, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n3,s3):6,(n5,s5):2,(n2,s2):7,(n4,s4):8LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ b9a165c03643ed43a0d45b16c0a795244543a2fe:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r833:/Table/106/1/7{583251934397083648-601680250155034624} [(n8,s8):6, (n4,s4):2, (n2,s2):4VOTER_DEMOTING_LEARNER, (n7,s7):7VOTER_INCOMING, next=8, gen=46, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):6,(n4,s4):2,(n2,s2):4VOTER_DEMOTING_LEARNER,(n7,s7):7VOTER_INCOMING
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r833:/Table/106/1/7{583251934397083648-601680250155034624} [(n8,s8):6, (n4,s4):2, (n2,s2):4VOTER_DEMOTING_LEARNER, (n7,s7):7VOTER_INCOMING, next=8, gen=46, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):6,(n4,s4):2,(n2,s2):4VOTER_DEMOTING_LEARNER,(n7,s7):7VOTER_INCOMING
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 33d70998719051ee058bc9e516afa238ea7b7451:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r894:/Table/106/1/1{01355736668733440-19784052426684416} [(n1,s1):11, (n4,s4):10, (n7,s7):12, (n6,s6):13LEARNER, next=14, gen=85, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):11,(n4,s4):10,(n7,s7):12,(n6,s6):13LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r894:/Table/106/1/1{01355736668733440-19784052426684416} [(n1,s1):11, (n4,s4):10, (n7,s7):12, (n6,s6):13LEARNER, next=14, gen=85, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):11,(n4,s4):10,(n7,s7):12,(n6,s6):13LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 03d5260a467b8640de41ee08054c1ce05f91966c:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r933:/Table/106/1/31{42027836730746880-60456152488699904} [(n8,s8):8, (n6,s6):10, (n4,s4):9, (n1,s1):11LEARNER, next=12, gen=93, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):8,(n6,s6):10,(n4,s4):9,(n1,s1):11LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r933:/Table/106/1/31{42027836730746880-60456152488699904} [(n8,s8):8, (n6,s6):10, (n4,s4):9, (n1,s1):11LEARNER, next=12, gen=93, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):8,(n6,s6):10,(n4,s4):9,(n1,s1):11LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 88789feea153b61b379c0240a07482bdfaf0c932:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r261:/Table/106/1/71{77828987722147840-96257303480098816} [(n6,s6):7, (n8,s8):2, (n2,s2):8VOTER_DEMOTING_LEARNER, (n3,s3):9VOTER_INCOMING, next=10, gen=72, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):7,(n8,s8):2,(n2,s2):8VOTER_DEMOTING_LEARNER,(n3,s3):9VOTER_INCOMING
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r261:/Table/106/1/71{77828987722147840-96257303480098816} [(n6,s6):7, (n8,s8):2, (n2,s2):8VOTER_DEMOTING_LEARNER, (n3,s3):9VOTER_INCOMING, next=10, gen=72, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):7,(n8,s8):2,(n2,s2):8VOTER_DEMOTING_LEARNER,(n3,s3):9VOTER_INCOMING
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 6374bd83f192bd509e50313acb431690b9c3f8ae:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r690:/Table/106/1/35{10594151889780736-29022467647731712} [(n7,s7):9, (n4,s4):10VOTER_DEMOTING_LEARNER, (n2,s2):5, (n6,s6):11VOTER_INCOMING, next=12, gen=74, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):9,(n4,s4):10VOTER_DEMOTING_LEARNER,(n2,s2):5,(n6,s6):11VOTER_INCOMING
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r690:/Table/106/1/35{10594151889780736-29022467647731712} [(n7,s7):9, (n4,s4):10VOTER_DEMOTING_LEARNER, (n2,s2):5, (n6,s6):11VOTER_INCOMING, next=12, gen=74, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n7,s7):9,(n4,s4):10VOTER_DEMOTING_LEARNER,(n2,s2):5,(n6,s6):11VOTER_INCOMING
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 86e007dcb5cbde3501f138c1d768519db3487857:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r177:/Table/106/1/17{41475839126425600-59904154884378624} [(n6,s6):16, (n1,s1):17, (n4,s4):15, (n3,s3):18LEARNER, next=19, gen=93, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):16,(n1,s1):17,(n4,s4):15,(n3,s3):18LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r177:/Table/106/1/17{41475839126425600-59904154884378624} [(n6,s6):16, (n1,s1):17, (n4,s4):15, (n3,s3):18LEARNER, next=19, gen=93, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n6,s6):16,(n1,s1):17,(n4,s4):15,(n3,s3):18LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 571bfa3afb3858ae84d8a8fcdbb0a38e058402a5:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n1,s1}]: unable to find store 8 in range r529:/Table/106/1/65{32837936193843200-51266251951794176} [(n7,s7):8, (n1,s1):10, (n3,s3):7, next=11, gen=54, sticky=9223372036.854775807,2147483647]
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n1,s1}]: unable to find store 8 in range r529:/Table/106/1/65{32837936193843200-51266251951794176} [(n7,s7):8, (n1,s1):10, (n3,s3):7, next=11, gen=54, sticky=9223372036.854775807,2147483647]
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 687171ac6c2cd9992486bb3b8c9d252ac95ca1cd:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r666:/Table/106/1/52{24427517379278848-42855833137229824} [(n8,s8):10, (n2,s2):9, (n4,s4):11VOTER_DEMOTING_LEARNER, (n6,s6):12VOTER_INCOMING, next=13, gen=74, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):10,(n2,s2):9,(n4,s4):11VOTER_DEMOTING_LEARNER,(n6,s6):12VOTER_INCOMING
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r666:/Table/106/1/52{24427517379278848-42855833137229824} [(n8,s8):10, (n2,s2):9, (n4,s4):11VOTER_DEMOTING_LEARNER, (n6,s6):12VOTER_INCOMING, next=13, gen=74, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n8,s8):10,(n2,s2):9,(n4,s4):11VOTER_DEMOTING_LEARNER,(n6,s6):12VOTER_INCOMING
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ 88d3253301457ac57820e0f4a4fab8f74bf9f38b:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r316:/Table/106/1/35{29022467647731712-47450783405682688} [(n5,s5):8, (n8,s8):6, (n2,s2):9, (n7,s7):10LEARNER, next=11, gen=61, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n5,s5):8,(n8,s8):6,(n2,s2):9,(n7,s7):10LEARNER
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r316:/Table/106/1/35{29022467647731712-47450783405682688} [(n5,s5):8, (n8,s8):6, (n2,s2):9, (n7,s7):10LEARNER, next=11, gen=61, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n5,s5):8,(n8,s8):6,(n2,s2):9,(n7,s7):10LEARNER
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on master @ e9ee21860458d997a8155734dc608cfcd050ef24:

test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
	monitor.go:127,kv.go:861,kv.go:889,test_runner.go:896: monitor failure: monitor task failed: pq: internal error: range r337:/Table/106/1/73{25255513785761792-43683829543712768} [(n1,s1):13, (n7,s7):14VOTER_DEMOTING_LEARNER, (n8,s8):10, (n4,s4):15VOTER_INCOMING, next=16, gen=82, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):13,(n7,s7):14VOTER_DEMOTING_LEARNER,(n8,s8):10,(n4,s4):15VOTER_INCOMING
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:861
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:889
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: internal error: range r337:/Table/106/1/73{25255513785761792-43683829543712768} [(n1,s1):13, (n7,s7):14VOTER_DEMOTING_LEARNER, (n8,s8):10, (n4,s4):15VOTER_INCOMING, next=16, gen=82, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n1,s1):13,(n7,s7):14VOTER_DEMOTING_LEARNER,(n8,s8):10,(n4,s4):15VOTER_INCOMING
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@nvanbenschoten nvanbenschoten added the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 18, 2022
@nvanbenschoten
Copy link
Member

The diagnosis of this issue was in #79887 (comment).

This is still failing after #80918. It's unclear to me whether that is expected or not, or whether we're still waiting on #80205. @aayushshah15 do you know? Were you still planning to push #80205 over the finish line?

@github-actions
Copy link

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aayushshah15 aayushshah15

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. S-1 High impact: many users impacted, serious risk of high unavailability or data loss T-kv KV Team X-stale
Projects
None yet
Milestone
No milestone
6 participants
@jlinder @AlexTalks @tbg @nvanbenschoten @aayushshah15 @cockroach-teamcity