Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: follower-reads/survival=zone/locality=global/reads=strong failed #70818

Closed
cockroach-teamcity opened this issue Sep 28, 2021 · 1 comment · Fixed by #75739
Closed

roachtest: follower-reads/survival=zone/locality=global/reads=strong failed #70818

cockroach-teamcity opened this issue Sep 28, 2021 · 1 comment · Fixed by #75739
Assignees
@andreimatei @aayushshah15
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. skipped-test

Comments

@cockroach-teamcity
Copy link
Member

roachtest.follower-reads/survival=zone/locality=global/reads=strong failed with artifacts on master @ 80844e44f4f9d0e565c6e810bd4c782be61899bf:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=strong/run_1
	follower_reads.go:657,follower_reads.go:319,follower_reads.go:57,test_runner.go:777: too many intervals with more than 2 nodes with low follower read ratios: 6 intervals > 4 threshold. Bad intervals:
		interval 09:47:00-09:47:10: n1 ratio: 0.998 n2 ratio: 0.495 n3 ratio: 0.657 n4 ratio: 0.000 n5 ratio: 2.711 n6 ratio: 0.993 
		interval 09:47:50-09:48:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.758 n4 ratio: 0.000 n5 ratio: 2.704 n6 ratio: 1.003 
		interval 09:48:00-09:48:10: n1 ratio: 0.837 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 n5 ratio: 2.727 n6 ratio: 0.993 
		interval 09:48:10-09:48:20: n1 ratio: 0.770 n2 ratio: 1.000 n3 ratio: 0.628 n4 ratio: 0.000 n5 ratio: 2.750 n6 ratio: 0.996 
		interval 09:48:40-09:48:50: n1 ratio: 0.184 n2 ratio: 1.000 n3 ratio: 0.417 n4 ratio: 0.000 n5 ratio: 2.750 n6 ratio: 1.000 
		interval 09:49:30-09:49:40: n1 ratio: 0.443 n2 ratio: 1.000 n3 ratio: 0.156 n4 ratio: 0.000 n5 ratio: 2.727 n6 ratio: 1.000
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 28, 2021
@tbg tbg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 4, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/survival=zone/locality=global/reads=strong failed with artifacts on master @ a5158c4c93e9dca858b38ef7e94321f0a0a2b5c5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=strong/run_1
	follower_reads.go:270,follower_reads.go:72,test_runner.go:780: failed to get follower read counts: Get "http://34.138.4.173:26258/_status/vars": dial tcp 34.138.4.173:26258: connect: connection refused
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

andreimatei added a commit to andreimatei/cockroach that referenced this issue Jan 31, 2022
@andreimatei
sql: fix the sql.trace.stmt.enable_threshold cluster setting
13c3b7c 
This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in cockroachdb#72296.

Fixes cockroachdb#75716
Fixes cockroachdb#75715
Fixes cockroachdb#75714
Fixes cockroachdb#71244
Fixes cockroachdb#71126
Fixes cockroachdb#70818
Fixes cockroachdb#70191
Fixes cockroachdb#70011
Fixes cockroachdb#70010
Fixes cockroachdb#69952
Fixes cockroachdb#69951

Release note: None
@andreimatei andreimatei mentioned this issue Jan 31, 2022
Merged
andreimatei added a commit to andreimatei/cockroach that referenced this issue Jan 31, 2022
@andreimatei
sql: fix the sql.trace.stmt.enable_threshold cluster setting
e85fe73 
This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in cockroachdb#72296.

Fixes cockroachdb#75716
Fixes cockroachdb#75715
Fixes cockroachdb#75714
Fixes cockroachdb#71244
Fixes cockroachdb#71126
Fixes cockroachdb#70818
Fixes cockroachdb#70191
Fixes cockroachdb#70011
Fixes cockroachdb#70010
Fixes cockroachdb#69952
Fixes cockroachdb#69951

Release note: None
craig bot pushed a commit that referenced this issue Feb 1, 2022
@andreimatei @yuzefovich
Merge #75739 #75764
19a5d53 
75739: sql: fix the sql.trace.stmt.enable_threshold cluster setting  r=andreimatei a=andreimatei

This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in #72296.

Fixes #75716
Fixes #75715
Fixes #75714
Fixes #71244
Fixes #71126
Fixes #70818
Fixes #70191
Fixes #70011
Fixes #70010
Fixes #69952
Fixes #69951

Release note: None

75764: roachtest: add a variant of tpch_concurrency with no admission control r=yuzefovich a=yuzefovich

Informs: #74836.

Release note: None

Co-authored-by: Andrei Matei <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in f5d78e2 Feb 1, 2022
@craig craig bot closed this as completed in #75739 Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreimatei andreimatei

@aayushshah15 aayushshah15

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. skipped-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: fix the sql.trace.stmt.enable_threshold cluster setting andreimatei/cockroach
4 participants
@andreimatei @tbg @aayushshah15 @cockroach-teamcity