Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv/kvclient/kvcoord: TestTxnCoordSenderRetries failed #111492

Closed
cockroach-teamcity opened this issue Sep 29, 2023 · 4 comments · Fixed by #111526
Closed

kv/kvclient/kvcoord: TestTxnCoordSenderRetries failed #111492

cockroach-teamcity opened this issue Sep 29, 2023 · 4 comments · Fixed by #111526
Assignees
Labels
branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 29, 2023

kv/kvclient/kvcoord.TestTxnCoordSenderRetries failed with artifacts on master @ d89e0f438dfd8fdd1e891bd222f88205458966f4:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:222 +0x2b6
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc00253fa18, {0xb2de528, 0xc000cb7b00}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:150 +0x1e7
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc00253f9e0, {0xb2de528, 0xc000cb7b00}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:146 +0xa06
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc00253f8b0, {0xb2de528, 0xc000cb7b00}, 0x1?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:295 +0x246
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc00253f890, {0xb2de528, 0xc000cb7b00}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:114 +0x3a6
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc00253f7e0, {0xb2de528, 0xc000cb7b00}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x7d4
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc00253f600, {0xb2de4b8, 0xc00013c008}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:533 +0xa33
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc00bbaae60, {0xb2de4b8, 0xc00013c008}, 0xc00531aea0, {0x7f6360345b28, 0xc00253f600})
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1090 +0x191
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc002578000, {0xb2de4b8, 0xc00013c008}, 0xc00531aea0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1218 +0x2e9
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0xb2de4b8, 0xc00013c008}, 0xc006e68f78, 0xc003c1ac00)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:922 +0x253
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run(0xc002578000, {0xb2de4b8, 0xc00013c008}, 0x73a9260?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:764 +0xa5
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).scan(0xc002578000, {0xb2de4b8, 0xc00013c008}, {0x73a9260, 0xb298de0}, {0x73a9260, 0xb298e30}, 0x0, 0x0?, 0x2, ...)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:610 +0x22d
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).ScanForShare(...)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:652
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestTxnCoordSenderRetries.func103({0xb2de4b8, 0xc00013c008}, 0x0?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_server_test.go:3081 +0x6f
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestTxnCoordSenderRetries.func206.1({0xb2de4b8, 0xc00013c008}, 0x0?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_server_test.go:4223 +0x35f
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc002578000, {0xb2de4b8, 0xc00013c008}, 0xc006e698c0)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1012 +0x9a
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.runTxn({0xb2de4b8, 0xc00013c008}, 0xc00013c008?, 0x9306900?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1055 +0x6c
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0xc00bbaae60, {0xb2de4b8, 0xc00013c008}, 0xa237f40?, 0xc0?, 0x0?, 0xc008212d80?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1018 +0xf1
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:993
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestTxnCoordSenderRetries.func206(0xc0076341a0, {{0x7db48a4, 0x33}, 0x0, 0x7ff88d8, 0x7ff88e0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_server_test.go:4187 +0x897
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestTxnCoordSenderRetries.func207.1(0x0?, 0x0?)
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_server_test.go:4256 +0xbe
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/isolation.RunEachLevel[...].func1()
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/isolation/pkg/kv/kvserver/concurrency/isolation/levels.go:76 +0x55
        	            	  | testing.tRunner(0xc0076341a0, 0xc00d9c27e0)
        	            	  | 	GOROOT/src/testing/testing.go:1576 +0x217
        	            	  | created by testing.(*T).Run
        	            	  | 	GOROOT/src/testing/testing.go:1629 +0x806
        	            	Error types: (1) *errbase.opaqueWrapper (2) *errutil.leafError
        	Test:       	TestTxnCoordSenderRetries/write_too_old_with_(replicated)_shared_locking_read/Serializable
        --- FAIL: TestTxnCoordSenderRetries/write_too_old_with_(replicated)_shared_locking_read/Serializable (0.18s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-31930

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Sep 29, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 29, 2023
@kvoli
Copy link
Collaborator

kvoli commented Sep 29, 2023

This is flaking a fair bit under race.

@nvanbenschoten
Copy link
Member

Looks like a failure with replicated locks, which were added to the test yesterday. I'll take a look.

@nvanbenschoten nvanbenschoten self-assigned this Sep 29, 2023
@nvanbenschoten
Copy link
Member

cannot write undeclared span /Local/Lock"a"

@nvanbenschoten
Copy link
Member

This is the same failure as #111409.

craig bot pushed a commit that referenced this issue Sep 29, 2023
109638: sql: use the correct locking strength for FOR SHARE clause r=michae2 a=arulajmani

Previously, FOR SHARE and FOR KEY SHARE would use non-locking KV scans.
Now that the lock table supports shared locks, we can use lock.Shared as
the locking strength for KV scans. This patch does that, and in doing
so, wires up SHARED locks end to end.

By default, we turn of this functionality for serializable transactions.
Instead, it's gated behind a session setting called
`enable_shared_locking_for_serializable`.

Informs #91545

Release note (sql change): SELECT FOR SHARE and SELECT FOR UPDATE
previously did not acquire any locks. Users issuing these statements
would expect them to acquire shared locks (multiple readers allowed,
no writers though). This patch switches over the behavior to acquire
such read locks.

For serializable transactions, we default to the old behaviour, unless
the `enable_shared_locking_for_serializable` session setting is set
to true. We'll probably switch this behavior in the future, but for
now, given shared locks are a preview feature, we gate things behind
a session setting.

111441: goschedstats: reduce underloadedRunnablePerProcThreshold r=aadityasondhi a=sumeerbhola

By reducing this threshold, we more often sample the stats at 1ms intervals, which is desirable for admission control slot adjustment.

Fixes #111125

Epic: none

Release note: None

111466: ui: update default timescale to 1h r=maryliag a=maryliag

On the Metrics page, the default value was 10min, which was too small. This commit updates it to 1h to match the most selected value on the SQL Activity.

Fixes #96479

Release note: None

111521: roachprod: fix createTenantCertBundle script r=andy-kimball a=herkolategan

The `createTenantCertBundle` function has a script in it that has erroneous "+" characters which results in an error when trying to start a secure cluster. This change removes those characters.

Epic: None
Release Note: None

111526: kv/spanset: permit writes to lock table by shared lock latching configuration r=nvanbenschoten a=nvanbenschoten

Fixes #111409.
Fixes #111492.

This commit updates addLockTableSpans to account for the way that shared locking requests declare their latches. Even though they declare a "read" latch, they do so at max timestamp, which gives them sufficient isolation to write to the lock table without having to declare a write latch and be serialized with other shared lock acquisitions.

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>
Co-authored-by: sumeerbhola <[email protected]>
Co-authored-by: maryliag <[email protected]>
Co-authored-by: Herko Lategan <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in 2c17fb7 Sep 29, 2023
@kvoli kvoli added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Oct 2, 2023
THardy98 pushed a commit to THardy98/cockroach that referenced this issue Oct 6, 2023
…guration

Fixes cockroachdb#111409.
Fixes cockroachdb#111492.

This commit updates addLockTableSpans to account for the way that shared locking
requests declare their latches. Even though they declare a "read" latch, they do
so at max timestamp, which gives them sufficient isolation to write to the lock
table without having to declare a write latch and be serialized with other
shared lock acquisitions.

Release note: None
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Projects
No open projects
Archived in project
3 participants