Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

c2c/admission: hit observed raft log position less than per-stream lower bound panic during race run of TestTenantStreamingMultipleNodes #106589

Closed
msbutler opened this issue Jul 11, 2023 · 2 comments · Fixed by #107412
Closed

c2c/admission: hit observed raft log position less than per-stream lower bound panic during race run of TestTenantStreamingMultipleNodes #106589

msbutler opened this issue Jul 11, 2023 · 2 comments · Fixed by #107412
Assignees
@irfansharif
Labels
A-admission-control C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-kv KV Team

Comments

@msbutler
Copy link
Collaborator

msbutler commented Jul 11, 2023 •
edited
Loading

UPDATE: I have repro'd the failure on a cleaner branch here. See newer CI failure here.

ORIGINAL MESSAGE: See the CI failure here in a draft PR that attempts to unskip TestTenantStreamingMultiNodes under stress race. Full test log for this run here. I have seen this stack trace on master Extended CI, but have yet to reproduce this locally on my gce worker.

Full stack trace:

	[TestTenantStreamingMultipleNodes] === RUN   TestTenantStreamingMultipleNodes
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1823897493
    test_log_scope.go:81: use -show-logs to present logs inline
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1  observed raft log position less than per-stream lower bound (log-position=0/0 <= log-position=9/6600)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !goroutine 255859 [running]:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !runtime/debug.Stack()
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	GOROOT/src/runtime/debug/stack.go:24 +0x72
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc010fa16e0, {{{0xc010f040c0, 0x24}, {0x94dc09b, 0x1}, {0x94dc09a, 0x1}, {0x94dc09b, 0x1}}, 0x1770ab40ea3045ae, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:261 +0xb7
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xd499400, 0xc011b828a0}, 0x2, 0x4, 0x0, 0x0, {0x9603daa, 0x46}, {0xc039c01720, 0x2, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x6e6
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf({0xd499400, 0xc011b828a0}, {0x9603daa, 0x46}, {0xc039c01720, 0x2, 0x2})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:848 +0x9d
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowtokentracker.(*Tracker).Track(0xc04b7bade0, {0xd499400, 0xc011b828a0}, 0x0, 0xbd, {0xc0154167a0?, 0x4?})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95 +0x85b
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle.(*Handle).deductTokensForInner(0xc02ddcfc00, {0xd499400, 0xc011b828a0}, 0x0?, {0x1350ce80?, 0xc0fb967b30?}, 0xbd)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle/kvflowhandle.go:162 +0x394
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle.(*Handle).DeductTokensFor(0xc02ddcfc00, {0xd499400, 0xc011b828a0}, 0x7d?, {0x2c?, 0xc015416880?}, 0x1?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/kvflowhandle/kvflowhandle.go:141 +0x72
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.maybeDeductFlowTokens({0xd499400, 0xc0f155cf60}, {0xd4df900, 0xc02ddcfc00}, {0xc05ce6e2d0, 0x3, 0x3}, {0xc037f647e0, 0x3, 0x3})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:1051 +0x2e5
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.proposeBatch({0xd499400, 0xc0f155cf60}, {0x7fcccd588ca0, 0xc0efe8a880}, {0xd4b9258, 0xc11e1c02d0}, {0xc037f647e0, 0x3, 0x3}, {0xc05ce6e2d0, ...}, ...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:1028 +0x24f
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc0efe8b0d0, {0xd499400, 0xc0f155cf60}, {0xd4b9258, 0xc11e1c02d0})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:621 +0x1c97
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x0?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:804 +0xfd
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1(...)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:2042
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc0efe8a880, 0x1, 0xc015418760)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:2043 +0x3ae
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:801 +0x3a5
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757 +0x210
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc00396aa80, 0xd499400?)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:660 +0x1d0
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc007028e70, {0xd499400, 0xc006dd95f0}, {0xd47cc40, 0xc00396aa80}, 0xc004303500)
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x2f6
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xd499400, 0xc006dd95f0})
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x9b
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f7
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x61a
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !For more context, check log files in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1823897493
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !****************************************************************************
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !This node experienced a fatal error (printed above), and as a result the
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !process is terminating.
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !will try to determine the root cause, recommend next steps, and we can
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !improve CockroachDB based on your report.
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !Please submit a crash report by following the instructions here:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !If you would rather not post publicly, please contact us directly at:
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !    [email protected]
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !
F230711 01:15:55.559224 255859 kv/kvserver/kvflowcontrol/kvflowtokentracker/tracker.go:95  [T1,n2,tenant=10,s2,r76/5:/Tenant/10/Table/1{1/2…-5/4…}] 1 !The Cockroach Labs team appreciates your feedback.

Jira issue: CRDB-29634
@msbutler msbutler added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-admission-control T-disaster-recovery labels Jul 11, 2023
@blathers-crl
Copy link

blathers-crl bot commented Jul 11, 2023

cc @cockroachdb/disaster-recovery

@msbutler msbutler added C-test-failure Broken test (automatically or manually discovered). T-kv KV Team labels Jul 11, 2023
@exalate-issue-sync exalate-issue-sync bot added sync-me and removed T-kv KV Team labels Jul 11, 2023
@irfansharif
Copy link
Contributor

Saw this earlier too: #106123.

@knz knz mentioned this issue Jul 19, 2023
Closed
@irfansharif irfansharif mentioned this issue Jul 22, 2023
Merged
craig bot pushed a commit that referenced this issue Jul 24, 2023
@irfansharif
Merge #107412
24603c9 
107412: kvserver: fix 'observed raft log position' assertion r=irfansharif a=irfansharif

Fixes #107336.
Fixes #106123.
Fixes #107156.
Fixes #106589.

It's possible to hit this assertion under --stress --race when the proposing replica is starved enough for raft ticks that it loses leadership right when it steps proposals through raft. We're relying on undocumented API semantics in the etcd raft library whereby it mutates stepped entries with the term+index its to end up in. But that's only applicable if stepping through entries as a leader. Simply relax this assertion instead.

Release note: None

Co-authored-by: irfan sharif <[email protected]>
@craig craig bot closed this as completed in #107412 Jul 24, 2023
@craig craig bot closed this as completed in deb538f Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
A-admission-control C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-kv KV Team
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: fix 'observed raft log position' assertion irfansharif/cockroach
3 participants
@irfansharif @msbutler @shermanCRL