Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/ccl/serverccl/statusccl/statusccl_test: TestTenantStatusAPI failed #99770

Closed
cockroach-teamcity opened this issue Mar 28, 2023 · 16 comments · Fixed by #106551
Closed

pkg/ccl/serverccl/statusccl/statusccl_test: TestTenantStatusAPI failed #99770

cockroach-teamcity opened this issue Mar 28, 2023 · 16 comments · Fixed by #106551
Assignees
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 28, 2023

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 3551dc089ac3efcb3772df2109164dcb01e79d8f:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI249676458
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI249676458
--- FAIL: TestTenantStatusAPI (39.87s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (1.35s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:222: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:222
        	Error:      	Not equal: 
        	            	expected: 889
        	            	actual  : 892
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.67s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-observability @cockroachdb/multi-tenant

This test on roachdash | Improve this report!

Jira issue: CRDB-26064

@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 28, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 28, 2023
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ a34af06d8e7230e4052db80d55da21a06013f81b:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI237475937
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI237475937
--- FAIL: TestTenantStatusAPI (39.39s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (1.09s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:222: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:222
        	Error:      	Not equal: 
        	            	expected: 967
        	            	actual  : 968
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.70s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ aec78f33d45a8376a0ecec885688bae60dbfb85c:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI294730777
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI294730777
--- FAIL: TestTenantStatusAPI (42.28s)
=== RUN   TestTenantStatusAPI/txn_id_resolution
=== CONT  TestTenantStatusAPI/txn_id_resolution
    tenant_status_test.go:1223: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1223
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71
        	            				github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:59
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41
        	            				pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1210
        	            				pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1241
        	Error:      	Should not be: 0x0
        	Test:       	TestTenantStatusAPI/txn_id_resolution
    --- FAIL: TestTenantStatusAPI/txn_id_resolution (0.05s)
=== RUN   TestTenantStatusAPI/txn_id_resolution/tenant_cluster
=== CONT  TestTenantStatusAPI/txn_id_resolution/tenant_cluster
    testing.go:1343: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
        --- FAIL: TestTenantStatusAPI/txn_id_resolution/tenant_cluster (0.02s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@j82w j82w assigned zachlite and unassigned j82w Apr 6, 2023
craig bot pushed a commit that referenced this issue Apr 10, 2023
100817: statusccl: fix flaky TestTenantStatusAPI txn_id_resolution r=j82w a=j82w

The txn resolution add a txn id to the cache, and 
then in a separate call updates the txn id with the 
fingerprint. The test previously assumed if there 
was a record it would always have a valid fingerprint, 
but that is not true. The test is modified to add retry 
logic to wait for the fingerprint to be updated. The 
retry time was set to always be 1 minute, because  
SucceedsSoon only waits 5 seconds for race scenario 
which is not enough time for the resolution to complete.

Epic: none
part of: #99770

Release note: none

100943: licenses: Update BSL change date for master/23/2 r=e-mbrown a=e-mbrown

This commit updates the BSL license for master/23.2.

Epic: REL-286
Release note: None

Co-authored-by: j82w <[email protected]>
Co-authored-by: e-mbrown <[email protected]>
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 82fddc52747a86c696a1f77a85abc25d05858cc1:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2157748142
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2157748142
--- FAIL: TestTenantStatusAPI (81.78s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 869
        	            	actual  : 877
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.20s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.39s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue May 2, 2023
102726: sql: fixes flaky TestSQLStatsCompactor r=j82w a=j82w

The TestSQLStatsCompactor validates the amount of scans done by the
compactor test. The new activity update job will also do a scan on the
stats table which can cause the count to be off. This disables the
update job to avoid any conflict. The test ran for 15k run under stress
with deadlock detection without any failures. 

tenant_span_stats test is being skipped under deadlock, because it
causes a race condition resulting in a flaky.

Epic: none
closes: #102022
Informs: #99770

Release note: None

Co-authored-by: j82w <[email protected]>
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ fbea48d5cd399ab1889ae8ce76b11f521d914686:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI826981125
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI826981125
--- FAIL: TestTenantStatusAPI (97.84s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (1.06s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 1006
        	            	actual  : 1005
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.67s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ fbea48d5cd399ab1889ae8ce76b11f521d914686:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2776330538
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2776330538
--- FAIL: TestTenantStatusAPI (15.63s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.05s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 936
        	            	actual  : 935
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.03s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 26713f81a397d13db4f2257eaa0f6bb70d1a7cb3:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI367343596
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI367343596
--- FAIL: TestTenantStatusAPI (74.38s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.42s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 829
        	            	actual  : 830
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.21s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 4226a83871bbce776bc9389fca5cf084b4bb7632:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI1832305128
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI1832305128
--- FAIL: TestTenantStatusAPI (72.69s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.50s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 823
        	            	actual  : 824
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.24s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 4226a83871bbce776bc9389fca5cf084b4bb7632:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI663993116
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI663993116
--- FAIL: TestTenantStatusAPI (72.76s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.43s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 820
        	            	actual  : 822
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.22s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ b4533bdbc4b478f0ad311bad80b62bd072cf61cf:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2549049034
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2549049034
--- FAIL: TestTenantStatusAPI (72.34s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.52s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 815
        	            	actual  : 816
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.24s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ b4533bdbc4b478f0ad311bad80b62bd072cf61cf:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2745110912
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI2745110912
--- FAIL: TestTenantStatusAPI (74.18s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.57s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 826
        	            	actual  : 828
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.30s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 4c24848334fa068ae0eef950c687b42497bb7bd0:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI3318769737
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:141: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI3318769737
--- FAIL: TestTenantStatusAPI (72.59s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.62s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:247: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:247
        	Error:      	Not equal: 
        	            	expected: 824
        	            	actual  : 825
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.24s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ a84cfb9d8fb0f3da6b0b9c4197caa4e79e2ca968:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI3666691168
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:142: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI3666691168
--- FAIL: TestTenantStatusAPI (16.69s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.05s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:248: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:248
        	Error:      	Not equal: 
        	            	expected: 1002
        	            	actual  : 1001
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.03s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ cff1bce2e6bb6aaf6c3619765749fc31af4de5b3:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI506876398
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:142: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/8bc2de686563d04c3522b0df244fc129/logTestTenantStatusAPI506876398
--- FAIL: TestTenantStatusAPI (76.16s)
=== RUN   TestTenantStatusAPI/txn_id_resolution
=== CONT  TestTenantStatusAPI/txn_id_resolution
    tenant_status_test.go:1314: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1314
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71
        	            				github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:59
        	            				github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41
        	            				pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1301
        	            				pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:1332
        	Error:      	Should not be: 0x0
        	Test:       	TestTenantStatusAPI/txn_id_resolution
    --- FAIL: TestTenantStatusAPI/txn_id_resolution (0.09s)
=== RUN   TestTenantStatusAPI/txn_id_resolution/tenant_cluster
=== CONT  TestTenantStatusAPI/txn_id_resolution/tenant_cluster
    testing.go:1343: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
        --- FAIL: TestTenantStatusAPI/txn_id_resolution/tenant_cluster (0.03s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rharding6373
Copy link
Collaborator

I've also seen this failure in CI: https://teamcity.cockroachdb.com/viewLog.html?buildId=10567081&buildTypeId=Cockroach_BazelEssentialCi

Failed
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:248: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:248
        	Error:      	Not equal: 
        	            	expected: 1013
        	            	actual  : 1033
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.05s)

zachlite pushed a commit to zachlite/cockroach that referenced this issue Jun 20, 2023
craig bot pushed a commit that referenced this issue Jun 20, 2023
100083: kvserver: record metrics for ErrProposalDropped r=pavelkalinnikov a=tbg

Touches #100096.

Epic: none
Release note: None


105093: sql: use datum alloc for crdb_internal stmt stats rows r=dt a=dt

Happened to observe a cluster running a customer test suite which included a query that inspected stmt stats often, causing the CRDB node to spend a considerable amount of CPU time in production of the stmt stats vtable, in particular allocating (and then GC'ing) individual datums, especially given how wide this table has become with the addition of storage stats.

This change uses a datum allocator to produce those rows to reduce the number of separate allocations from the runtime.

Release note: none.
Epic: none.

105197: statusccl: skip flaky TenantStatusAPI tests r=zachlite a=zachlite

Informs #92382, #99770, #99559
Epic: none
Release note: None

Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: David Taylor <[email protected]>
Co-authored-by: Zach Lite <[email protected]>
blathers-crl bot pushed a commit that referenced this issue Jun 20, 2023
Informs #92382, #99770, #99559
Epic: none
Release note: None
zachlite pushed a commit to zachlite/cockroach that referenced this issue Jun 20, 2023
@zachlite zachlite reopened this Jul 5, 2023
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI failed with artifacts on release-23.1 @ 503ea5248f37ee180a6c82ece0a41bf05c833407:

=== RUN   TestTenantStatusAPI
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/7067ad80d000d85ce2eeee021e911c58/logTestTenantStatusAPI1776511918
=== CONT  TestTenantStatusAPI
    tenant_status_test.go:145: -- test log scope end --
CREATE UNIQUE INDEX test_pkey ON test_db1.public.test USING btree (k ASC)test logs left over in: /artifacts/tmp/_tmp/7067ad80d000d85ce2eeee021e911c58/logTestTenantStatusAPI1776511918
--- FAIL: TestTenantStatusAPI (24.77s)
=== RUN   TestTenantStatusAPI/tenant_span_stats
    --- FAIL: TestTenantStatusAPI/tenant_span_stats (0.05s)
=== RUN   TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
    tenant_status_test.go:251: 
        	Error Trace:	pkg/ccl/serverccl/statusccl/statusccl_test/pkg/ccl/serverccl/statusccl/tenant_status_test.go:251
        	Error:      	Not equal: 
        	            	expected: 1095
        	            	actual  : 1094
        	Test:       	TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out
        --- FAIL: TestTenantStatusAPI/tenant_span_stats/test_KV_node_fan-out (0.03s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

zachlite added a commit to zachlite/cockroach that referenced this issue Jul 10, 2023
This commit fixes flaky behavior while running
`TestSpanStatsFanOut` and `TestTenantSpanStats` under stress.

Both tests have been updated to ensure the following behavior:
- The tests make sure range splits occur before proceeding.
- The tests will retry their assertions to give the new key-value pairs
time to replicate.

Additionally, `NewTestTenantHelper` was updated to accept a parameter
for the number of nodes in the test host cluster. `TestTenantSpanStats`
now uses a 3-node cluster to test a real fan-out.

Resolves cockroachdb#99770
Resolves cockroachdb#99559
Epic:none
Release note: None
craig bot pushed a commit that referenced this issue Jul 11, 2023
106551: server, statusccl: de-flake span stats fan-out tests r=zachlite a=zachlite

This commit fixes flaky behavior while running
`TestSpanStatsFanOut` and `TestTenantSpanStats` under stress.

Both tests have been updated to ensure the following behavior:
- The tests make sure range splits occur before proceeding.
- The tests will retry their assertions to give the new key-value pairs time to replicate.

Additionally, `NewTestTenantHelper` was updated to accept a parameter for the number of nodes in the test host cluster. `TestTenantSpanStats` now uses a 3-node cluster to test a real fan-out.

Resolves #99770
Resolves #99559
Epic:none
Release note: None

106627: diagnosticsccl: unskip TestTenantReport r=yuzefovich a=yuzefovich

The flakiness of the test was resolved by #106053.

Addresses: #101622.
Epic: None

Release note: None

Co-authored-by: zachlite <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in f1bc66e Jul 11, 2023
blathers-crl bot pushed a commit that referenced this issue Jul 11, 2023
This commit fixes flaky behavior while running
`TestSpanStatsFanOut` and `TestTenantSpanStats` under stress.

Both tests have been updated to ensure the following behavior:
- The tests make sure range splits occur before proceeding.
- The tests will retry their assertions to give the new key-value pairs
time to replicate.

Additionally, `NewTestTenantHelper` was updated to accept a parameter
for the number of nodes in the test host cluster. `TestTenantSpanStats`
now uses a 3-node cluster to test a real fan-out.

Resolves #99770
Resolves #99559
Epic:none
Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants