Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestBackpressureNotAppliedWhenReducingRangeSize failed #75609

Closed
cockroach-teamcity opened this issue Jan 27, 2022 · 9 comments · Fixed by #75612
Closed

kv/kvserver: TestBackpressureNotAppliedWhenReducingRangeSize failed #75609

cockroach-teamcity opened this issue Jan 27, 2022 · 9 comments · Fixed by #75612
Assignees
@irfansharif
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize824066524
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize824066524
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (75.86s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 45s: could not find replica
        goroutine 41407012 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x5413e78, 0xc01fd32000}, 0x0, 0xc017014540)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x5413e78, 0xc01fd32000}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc01fd32000, {0x54f3fd0, 0xc002046580}, {0xc016dda458, 0x1, 0x8})
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_learner_test.go:97 +0x167
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0xda
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x4a
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x4783e0, 0xc00ac92be0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xc2
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc00710cce0, 0x3c5abe0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x74
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x5413e78, 0xc01fd32000}, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x47
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x5413e78, 0xc01fd32000}, 0x54b1cc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0x534e7e0, 0xc01e7de240, {0xc016dda458, 0x0, 0x0}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0x7b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc01fd32000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x4af
        testing.tRunner(0xc01fd32000, 0xc01a22a1c0)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (52.66s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=2

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 27, 2022
@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize3894882914
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize3894882914
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (264.41s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 3m45s: could not find replica
        goroutine 478233 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b6b8, 0xc0063b2b60}, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b6b8, 0xc0063b2b60}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc0063b2b60, {0x92776d0, 0xc002fe2580}, {0xc006118058, 0x1, 0x8})
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_learner_test.go:97 +0x170
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0x148
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x62
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x0, 0xc001680940)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xd1
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc0041cea50, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x85
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b6b8, 0xc0063b2b60}, 0x4981d9, 0x499837)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b6b8, 0xc0063b2b60}, 0x9235f00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0xc0063b2b60, 0xc002fe2000, {0xc006118058, 0x1, 0x8}, 0x38182)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0xc9
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc0063b2b60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x629
        testing.tRunner(0xc0063b2b60, 0xc00aef2040)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (236.08s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-race -parallel=2

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize3943585046
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize3943585046
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (272.11s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 3m45s: could not find replica
        goroutine 485530 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b6b8, 0xc006b6a000}, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b6b8, 0xc006b6a000}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc006b6a000, {0x92776d0, 0xc006335b80}, {0xc00b9bc098, 0x1, 0x8})
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_learner_test.go:97 +0x170
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0x148
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x62
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x0, 0xc001a0c940)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xd1
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc0040c2a50, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x85
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b6b8, 0xc006b6a000}, 0x4981d9, 0x499837)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b6b8, 0xc006b6a000}, 0x9235f00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0xc006b6a000, 0xc006334580, {0xc00b9bc098, 0x1, 0x8}, 0x38184)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0xc9
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc006b6a000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x629
        testing.tRunner(0xc006b6a000, 0xc00aeba000)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (238.10s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ 223b121d4e362470b2405eb3ccc59e0d21c669e2:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize2850686271
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize2850686271
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (272.83s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 3m45s: could not find replica
        goroutine 493391 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b798, 0xc0013b2340}, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b798, 0xc0013b2340}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc0013b2340, {0x92777b0, 0xc008675600}, {0xc000a06378, 0x1, 0x8})
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_learner_test.go:97 +0x170
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0x148
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x62
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x0, 0xc000bf2940)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xd1
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc005d2ca50, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x85
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919b798, 0xc0013b2340}, 0x4981d9, 0x499837)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919b798, 0xc0013b2340}, 0x9235fe0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0xc0013b2340, 0xc008674580, {0xc000a06378, 0x1, 0x8}, 0x3817f)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0xc9
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc0013b2340)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x629
        testing.tRunner(0xc0013b2340, 0xc004710000)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (238.93s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ c2023effaad7966ace04f4d84170f96dc47ce6ba:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestBackpressureNotAppliedWhenReducingRangeSize3590271314
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (75.50s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 45s: could not find replica
        goroutine 403221 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x543ed98, 0xc009040000}, 0x0, 0xc003dc7840)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x543ed98, 0xc009040000}, 0x0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc009040000, {0x5518bc8, 0xc002320580}, {0xc00b13eae0, 0x1, 0x8})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/replica_learner_test.go:97 +0x167
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0xda
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x4a
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x478740, 0xc002d92be0)
        	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xc2
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc006046ce0, 0x3c85720)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x74
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x543ed98, 0xc009040000}, 0x0, 0x0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x47
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x543ed98, 0xc009040000}, 0x54d8740)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0x5380640, 0xc008462b40, {0xc00b13eae0, 0x0, 0x0}, 0x0)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0x7b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc009040000)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x4af
        testing.tRunner(0xc009040000, 0xc005088000)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (52.94s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss

Same failure on other branches

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Jan 27, 2022

seems pretty broken, bisecting

@irfansharif irfansharif self-assigned this Jan 27, 2022
@irfansharif
Copy link
Contributor

irfansharif commented Jan 27, 2022 •
edited
Loading

@tbg I was in this area recently (#75598), I'll take a look.

@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ c2023effaad7966ace04f4d84170f96dc47ce6ba:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestBackpressureNotAppliedWhenReducingRangeSize282220933
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (75.56s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 45s: could not find replica
        goroutine 43479323 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x544c2b8, 0xc000c741a0}, 0x0, 0xc00c386240)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x544c2b8, 0xc000c741a0}, 0x0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc000c741a0, {0x552c208, 0xc00448b080}, {0xc0298e6490, 0x1, 0x8})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/replica_learner_test.go:97 +0x167
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0xda
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x4a
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x478740, 0xc012e60be0)
        	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xc2
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc00cee8ce0, 0x3c8f500)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x74
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x544c2b8, 0xc000c741a0}, 0x0, 0x0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x47
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x544c2b8, 0xc000c741a0}, 0x54ea200)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0x5386c40, 0xc011198000, {0xc0298e6490, 0x0, 0x0}, 0x0)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0x7b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc000c741a0)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x4af
        testing.tRunner(0xc000c741a0, 0xc021f36040)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (52.55s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Jan 27, 2022

Okay, sounds good, fwiw [b64fe37] Merge #75550 #75573 seems good

@irfansharif irfansharif mentioned this issue Jan 27, 2022
Merged
irfansharif added a commit to irfansharif/cockroach that referenced this issue Jan 27, 2022
@irfansharif
kvserver: deflake a deflake attempt for ..
05887bb 
...TestBackpressureNotAppliedWhenReducingRangeSize. Fixes cockroachdb#75609, a
regression we introduced in cockroachdb#75598. When waiting for a span config to
materialize on a given node, we were accidentally tripping up on it not
being found on the original. Not sure we this slipped CI.

Release note: None
@irfansharif irfansharif mentioned this issue Jan 27, 2022
Merged
@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize failed with artifacts on master @ 819cc5e7a0642ebbb2872541cd227c79c5025d75:

=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize84748222
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestBackpressureNotAppliedWhenReducingRangeSize
    client_replica_backpressure_test.go:313: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBackpressureNotAppliedWhenReducingRangeSize84748222
--- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize (273.25s)
=== RUN   TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node
    client_replica_backpressure_test.go:134: condition failed to evaluate within 3m45s: could not find replica
        goroutine 487910 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919f058, 0xc009868000}, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919f058, 0xc009868000}, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.getFirstStoreReplica(0xc009868000, {0x927b070, 0xc001288b00}, {0xc00cd3f8b8, 0x1, 0x8})
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_learner_test.go:97 +0x170
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2.1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:134 +0x148
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71 +0x62
        github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0x0, 0xc001586940)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207 +0xd1
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc00bd8ca50, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77 +0x85
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x919f058, 0xc009868000}, 0x4981d9, 0x499837)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:58 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x919f058, 0xc009868000}, 0x92398a0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func2(0xc009868000, 0xc001288000, {0xc00cd3f8b8, 0x1, 0x8}, 0x3817d)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:131 +0xc9
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestBackpressureNotAppliedWhenReducingRangeSize.func7(0xc009868000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_backpressure_test.go:272 +0x629
        testing.tRunner(0xc009868000, 0xc00a486000)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    --- FAIL: TestBackpressureNotAppliedWhenReducingRangeSize/backpressure_when_near_limit_on_new_node (238.52s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

This test on roachdash | Improve this report!

irfansharif added a commit to irfansharif/cockroach that referenced this issue Jan 27, 2022
@irfansharif
kvserver: deflake a deflake attempt for ..
e7dcb09 
...TestBackpressureNotAppliedWhenReducingRangeSize. Fixes cockroachdb#75609, a
regression we introduced in cockroachdb#75598. When waiting for a span config to
materialize on a given node, we were accidentally tripping up on it not
being found on the original. Not sure we this slipped CI.

Release note: None
@irfansharif irfansharif mentioned this issue Jan 28, 2022
Closed
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: deflake a deflake attempt for .. irfansharif/cockroach
3 participants
@tbg @irfansharif @cockroach-teamcity