kv/kvserver: TestRestoreReplicas failed

[cockroach-teamcity]

kv/kvserver.TestRestoreReplicas failed with artifacts on master @ 63eb6b86c1518fbed9db10e7d2afa1446b2b3b82:

=== RUN   TestRestoreReplicas
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3865623463
    test_log_scope.go:79: use -show-logs to present logs inline
    client_raft_test.go:402: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:402
        	Error:      	Not equal: 
        	            	expected: 2
        	            	actual  : 1
        	Test:       	TestRestoreReplicas
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3865623463
--- FAIL: TestRestoreReplicas (12.31s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-27902

Epic CRDB-27234

[cockroach-teamcity]

kv/kvserver.TestRestoreReplicas failed with artifacts on master @ 63eb6b86c1518fbed9db10e7d2afa1446b2b3b82:

=== RUN   TestRestoreReplicas
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3808935525
    test_log_scope.go:79: use -show-logs to present logs inline
    client_raft_test.go:402: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:402
        	Error:      	Not equal: 
        	            	expected: 2
        	            	actual  : 1
        	Test:       	TestRestoreReplicas
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3808935525
--- FAIL: TestRestoreReplicas (4.52s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

kv/kvserver.TestRestoreReplicas failed with artifacts on master @ a65bca232dfde1862e5aff343e7b1f3a379d6cd6:

=== RUN   TestRestoreReplicas
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3103863021
    test_log_scope.go:79: use -show-logs to present logs inline
    client_raft_test.go:402: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:402
        	Error:      	Not equal: 
        	            	expected: 2
        	            	actual  : 1
        	Test:       	TestRestoreReplicas
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRestoreReplicas3103863021
--- FAIL: TestRestoreReplicas (5.22s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[aliher1911]

This test started failing after: 117dc40

Instead of single lease failure we get two now:

    client_raft_test.go:397: failed to increment store s1: [NotLeaseHolderError] refusing to acquire lease on follower; r64: replica (n1,s1):1 not lease holder; current lease is repl=(n2,s2):2 seq=0 start=0,0 exp=<nil>
    client_raft_test.go:397: failed to increment store s2: [NotLeaseHolderError] lease held by different store; r64: replica (n2,s2):2 not lease holder; current lease is repl=(n1,s1):1 seq=1 start=0,0 exp=1684145189.860061524,0 pro=1684145183.860061524,0

[aliher1911]

@erikgrinaker mind having a look? seems relevant to the changes in that commit that touches lease acquisitions.

[cockroach-teamcity]

kv/kvserver.TestRestoreReplicas failed with artifacts on master @ 19e57552fcfa8bb6f07945f699e67f3b8179be89:

=== RUN   TestRestoreReplicas
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/7fb080117a3da2d7cdf3576571d60489/logTestRestoreReplicas179368972
    test_log_scope.go:79: use -show-logs to present logs inline
    client_raft_test.go:402: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:402
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	TestRestoreReplicas
        	Messages:   	replica command failed (non leaseholders)
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/7fb080117a3da2d7cdf3576571d60489/logTestRestoreReplicas179368972
--- FAIL: TestRestoreReplicas (4.76s)

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!