roachtest: transfer-leases/quit failed [slow upreplication] #58492

cockroach-teamcity · 2021-01-06T12:58:12Z

(roachtest).transfer-leases/quit failed on master@cee475331ca3629b503cd2e7c7919b72c98a5ca5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/transfer-leases/quit/run_1
	quit.go:63,quit.go:303,soon.go:47,retry.go:197,soon.go:53,quit.go:216,quit.go:86,quit.go:142,context.go:140,quit.go:141,quit.go:86,quit.go:46,quit.go:340,test_runner.go:760: (1) ranges with no lease outside of node 3: []string{"21"}

	cluster.go:1637,context.go:140,cluster.go:1626,test_runner.go:841: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2561359-1609916496-73-n3cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 9751
		1: 13528
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1850
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /transfer-leases/quit

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

nvanbenschoten · 2021-01-06T16:57:23Z

@knz do you mind taking a look at this? It's unclear why node 3 was detected as dead, as I don't see any crashes in the logs.

knz · 2021-01-06T17:12:29Z

I recently found that the cluster monitor logic in roachtest may have a flaw. Maybe this is a smoking gun of that. Will investigate.

asubiotto · 2021-01-21T13:24:35Z

@knz want to double check whether this issue should qualify as an alpha release blocker as well

knz · 2021-01-21T13:28:19Z

definitely not release blocker IMHO -- lease transfers that don't fully succeed during node restarts yield merely temporary inconvenience, and the issue is not new anyway

knz · 2021-01-25T13:47:00Z

@tbg I investigated this failure. It is expecting all leases to move out of some node (here, n3) and asserts this by checking, for every replicas on the node, that the lease is elsewhere.

In this particular case, it appears that the lease for affected range r21 was not moved out of the draining node ... because the lease was invalid (cue in discussion on #58613). Which is expected.

Now I'd like to tweak the test so that it excludes ranges with invalid leases. How can I inspect whether leases are invalid using SQL?

tbg · 2021-02-04T09:06:03Z

Sorry it took me so long to get back to you here. To the best of my knowledge, this is not possible. Do you feel that this test needs to remain a roachtest? As a roachtest, I would expect it to set up a workload and assert that the workload is not negatively impacted by the node draining. (Which, btw, is also a pain to set up in a roachtest, as you know, and we already have that roachtest: kv/gracefuldraining). Moreover, is this test adding enough value to be kept around, if we don't feel that it should remain a roachtest, assuming we get #59371 in?

knz · 2021-02-04T09:10:42Z

I agree that the structure of test is redundant with the code in #59371

However that other PR does not yet assert that all the leases have been transferred by a drain. We probably want to test that still. What would be a good non-SQL way to inspect whether all non-invalid leases have been transferred, and only invalid leases remain?

tbg · 2021-02-04T09:22:35Z

That would be store.VisitReplicas which lets you look at all replicas, then you can grab the .Lease() and check it against node liveness.

cockroach-teamcity · 2021-03-24T08:04:39Z

(roachtest).transfer-leases/quit failed on master@9fa4b125bfb07552b43ba4fd52c9301afd7a937b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/transfer-leases/quit/run_1
	quit.go:63,quit.go:303,soon.go:47,retry.go:197,soon.go:53,quit.go:216,quit.go:86,quit.go:142,context.go:140,quit.go:141,quit.go:86,quit.go:46,quit.go:340,test_runner.go:768: (1) ranges with no lease outside of node 3: []string{"26"}

	cluster.go:1667,context.go:140,cluster.go:1656,test_runner.go:849: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2807998-1616565826-23-n3cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 24573
		1: 1266
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /transfer-leases/quit

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-07-28T11:43:54Z

roachtest.transfer-leases/quit failed with artifacts on master @ 8009ac6bb63f455a18bb1139d505bdd9930816b7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/transfer-leases/quit/run_1
	quit.go:67,quit.go:139,quit.go:73,quit.go:50,quit.go:347,test_runner.go:770: cluster did not up-replicate: some ranges not up-replicated yet

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

tbg · 2021-07-28T21:10:18Z

#68171 (comment)

cockroach-teamcity · 2021-07-29T11:53:03Z

roachtest.transfer-leases/quit failed with artifacts on master @ f19914b8c6281e463645580e1411774c3b0c20c9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/transfer-leases/quit/run_1
	quit.go:67,quit.go:139,quit.go:73,quit.go:50,quit.go:347,test_runner.go:770: cluster did not up-replicate: some ranges not up-replicated yet

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-30T11:22:37Z

roachtest.transfer-leases/quit failed with artifacts on master @ fe1fb73ae989142193643db30dd4b1b6dd6fe7dd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/transfer-leases/quit/run_1
	quit.go:67,quit.go:139,quit.go:73,quit.go:50,quit.go:347,test_runner.go:777: cluster did not up-replicate: some ranges not up-replicated yet

Reproduce

roachtest: transfer-leases/quit failed [slow upreplication] #58492

roachtest: transfer-leases/quit failed [slow upreplication] #58492

Comments

cockroach-teamcity commented Jan 6, 2021

nvanbenschoten commented Jan 6, 2021

knz commented Jan 6, 2021

asubiotto commented Jan 21, 2021

knz commented Jan 21, 2021

knz commented Jan 25, 2021

tbg commented Feb 4, 2021

knz commented Feb 4, 2021

tbg commented Feb 4, 2021

cockroach-teamcity commented Mar 24, 2021

cockroach-teamcity commented Jul 28, 2021

tbg commented Jul 28, 2021 via email

cockroach-teamcity commented Jul 29, 2021

cockroach-teamcity commented Jul 30, 2021

cockroach-teamcity commented Jul 31, 2021

cockroach-teamcity commented Aug 1, 2021

cockroach-teamcity commented Aug 2, 2021

cockroach-teamcity commented Aug 3, 2021

cockroach-teamcity commented Aug 4, 2021

cockroach-teamcity commented Aug 5, 2021

cockroach-teamcity commented Aug 6, 2021

cockroach-teamcity commented Aug 7, 2021

cockroach-teamcity commented Aug 9, 2021

cockroach-teamcity commented Aug 10, 2021

cockroach-teamcity commented Aug 13, 2021

cockroach-teamcity commented Aug 14, 2021

cockroach-teamcity commented Aug 15, 2021

cockroach-teamcity commented Aug 16, 2021

cockroach-teamcity commented Aug 17, 2021

erikgrinaker commented Aug 23, 2021