rac2: panic pull mode must not have sending entries (apply_to_all enabled)

[kvoli]

Seen on kv/splits/nodes=3/quiesce=false/lease=leader as part of #136215. Also see this (internal) slack link for more context.

E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678  a panic has occurred!
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +pull mode must not have sending entries
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +(1) attached stack trace
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  -- stack trace:
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | runtime.gopanic
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	GOROOT/src/runtime/panic.go:770
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | [...repeated from below...]
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +Wraps: (2) assertion failure
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +Wraps: (3) attached stack trace
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  -- stack trace:
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*replicaSendStream).handleReadyEntriesRaftMuAndStreamLocked
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:2657
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*replicaState).handleReadyEntriesRaftMuLocked
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:2411
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:1220
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go:802
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/replica_raft.go:1017
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/replica_raft.go:836
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/store_raft.go:682
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/scheduler.go:419
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/kv/kvserver/scheduler.go:319
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	pkg/util/stop/stopper.go:498
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | runtime.goexit
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +  | 	src/runtime/asm_amd64.s:1695
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +Wraps: (4) pull mode must not have sending entries
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError

logs above the panic appear leader-lease related:

I241127 04:24:52.189625 281 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77648/1:‹/Table/106/1/-7430{370…-140…}›,raft] 7608  rejected lease request repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.424455422,0 pro=1732681491.424455422,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.424455422,0 pro=1732681491.424455422,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.424455422,0 pro=1732681491.424455422,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.424455422,0 pro=1732681491.424455422,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.224641 287 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77605/1:‹/Table/106/1/-7291{791…-560…}›,raft] 7609  rejected lease request repl=(n2,s2):3 seq=7 start=1732681103.300889144,0 exp=1732681497.394267040,0 pro=1732681491.394267040,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681103.300889144,0 exp=1732681497.394267040,0 pro=1732681491.394267040,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681103.300889144,0 exp=1732681497.394267040,0 pro=1732681491.394267040,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681103.300889144,0 exp=1732681497.394267040,0 pro=1732681491.394267040,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.256096 298 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77647/1:‹/Table/106/1/-742{8065…-7834…}›,raft] 7610  rejected lease request repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681496.987267070,0 pro=1732681490.987267070,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681496.987267070,0 pro=1732681490.987267070,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681496.987267070,0 pro=1732681490.987267070,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681496.987267070,0 pro=1732681490.987267070,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.315895 274 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77626/1:‹/Table/106/1/-7217{544…-313…}›,raft] 7611  rejected lease request repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.367261701,0 pro=1732681491.367261701,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.367261701,0 pro=1732681491.367261701,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.367261701,0 pro=1732681491.367261701,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.367261701,0 pro=1732681491.367261701,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.324645 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77672/1:‹/Table/106/1/-11{3100…-2869…}›,raft] 7612  rejected lease request repl=(n2,s2):3 seq=9 start=1732681006.371139373,0 exp=1732681497.482985280,0 pro=1732681491.482985280,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681006.371139373,0 exp=1732681497.482985280,0 pro=1732681491.482985280,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681006.371139373,0 exp=1732681497.482985280,0 pro=1732681491.482985280,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681006.371139373,0 exp=1732681497.482985280,0 pro=1732681491.482985280,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.324903 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77610/1:‹/Table/106/1/-7487{785…-555…}›,raft] 7613  rejected lease request repl=(n2,s2):3 seq=5 start=1732680944.790397288,0 exp=1732681497.410023911,0 pro=1732681491.410023911,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732680944.790397288,0 exp=1732681497.410023911,0 pro=1732681491.410023911,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732680944.790397288,0 exp=1732681497.410023911,0 pro=1732681491.410023911,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732680944.790397288,0 exp=1732681497.410023911,0 pro=1732681491.410023911,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.325036 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77604/1:‹/Table/106/1/-7243{369…-138…}›,raft] 7614  rejected lease request repl=(n2,s2):3 seq=8 start=1732681005.973692023,0 exp=1732681497.460133436,0 pro=1732681491.460133436,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681005.973692023,0 exp=1732681497.460133436,0 pro=1732681491.460133436,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681005.973692023,0 exp=1732681497.460133436,0 pro=1732681491.460133436,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681005.973692023,0 exp=1732681497.460133436,0 pro=1732681491.460133436,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.325246 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77540/1:‹/Table/106/1/-7286{257…-026…}›,raft] 7615  rejected lease request repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.325338 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77540/1:‹/Table/106/1/-7286{257…-026…}›,raft] 7616  rejected lease request repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681490.798681867,0 exp=1732681497.143136025,0 pro=1732681491.143136025,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.325487 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77628/1:‹/Table/106/1/-7218{697…-466…}›,raft] 7617  rejected lease request repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.393296315,0 pro=1732681491.393296315,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.393296315,0 pro=1732681491.393296315,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.393296315,0 pro=1732681491.393296315,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680944.775854437,0 exp=1732681497.393296315,0 pro=1732681491.393296315,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.331968 275 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77646/1:‹/Table/106/1/-7424{606…-375…}›,raft] 7618  rejected lease request repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.406354636,0 pro=1732681491.406354636,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.406354636,0 pro=1732681491.406354636,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.406354636,0 pro=1732681491.406354636,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.406354636,0 pro=1732681491.406354636,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.340079 293 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77649/1:‹/Table/106/1/-7480{868…-637…}›,raft] 7619  rejected lease request repl=(n2,s2):3 seq=5 start=1732681103.703809330,0 exp=1732681497.358909954,0 pro=1732681491.358909954,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732681103.703809330,0 exp=1732681497.358909954,0 pro=1732681491.358909954,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732681103.703809330,0 exp=1732681497.358909954,0 pro=1732681491.358909954,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732681103.703809330,0 exp=1732681497.358909954,0 pro=1732681491.358909954,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.344427 290 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77625/1:‹/Table/106/1/-7451{584…-353…}›,raft] 7620  rejected lease request repl=(n2,s2):3 seq=6 start=1732681103.700938536,0 exp=1732681497.409648822,0 pro=1732681491.409648822,0 acq=Request; current lease repl=(n2,s2):3 seq=6 start=1732681103.700938536,0 exp=1732681497.409648822,0 pro=1732681491.409648822,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=6 start=1732681103.700938536,0 exp=1732681497.409648822,0 pro=1732681491.409648822,0 acq=Request with repl=(n2,s2):3 seq=6 start=1732681103.700938536,0 exp=1732681497.409648822,0 pro=1732681491.409648822,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.345067 286 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77653/1:‹/Table/106/1/-7415{613…-383…}›,raft] 7621  rejected lease request repl=(n2,s2):3 seq=6 start=1732681103.895527424,0 exp=1732681497.396078366,0 pro=1732681491.396078366,0 acq=Request; current lease repl=(n2,s2):3 seq=6 start=1732681103.895527424,0 exp=1732681497.396078366,0 pro=1732681491.396078366,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=6 start=1732681103.895527424,0 exp=1732681497.396078366,0 pro=1732681491.396078366,0 acq=Request with repl=(n2,s2):3 seq=6 start=1732681103.895527424,0 exp=1732681497.396078366,0 pro=1732681491.396078366,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.345192 289 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77645/1:‹/Table/106/1/-721{3163…-2932…}›,raft] 7622  rejected lease request repl=(n2,s2):3 seq=7 start=1732681103.711736359,0 exp=1732681497.441643329,0 pro=1732681491.441643329,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681103.711736359,0 exp=1732681497.441643329,0 pro=1732681491.441643329,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681103.711736359,0 exp=1732681497.441643329,0 pro=1732681491.441643329,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681103.711736359,0 exp=1732681497.441643329,0 pro=1732681491.441643329,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.345794 296 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77673/1:‹/Table/106/1/-735{6123…-5893…}›,raft] 7623  rejected lease request repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.478046935,0 pro=1732681491.478046935,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.478046935,0 pro=1732681491.478046935,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.478046935,0 pro=1732681491.478046935,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.478046935,0 pro=1732681491.478046935,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.345956 275 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77664/1:‹/Table/106/1/-74{1007…-0984…}›,raft] 7624  rejected lease request repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.518809943,0 pro=1732681491.518809943,0 acq=Request; current lease repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.518809943,0 pro=1732681491.518809943,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.518809943,0 pro=1732681491.518809943,0 acq=Request with repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.518809943,0 pro=1732681491.518809943,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.346016 296 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77607/1:‹/Table/106/1/-9017{693…-462…}›,raft] 7625  rejected lease request repl=(n2,s2):3 seq=7 start=1732681005.874462539,0 exp=1732681497.534355815,0 pro=1732681491.534355815,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681005.874462539,0 exp=1732681497.534355815,0 pro=1732681491.534355815,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681005.874462539,0 exp=1732681497.534355815,0 pro=1732681491.534355815,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681005.874462539,0 exp=1732681497.534355815,0 pro=1732681491.534355815,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.347320 296 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77609/1:‹/Table/106/1/-7298{478…-247…}›,raft] 7626  rejected lease request repl=(n2,s2):3 seq=7 start=1732680944.709853941,0 exp=1732681497.438292883,0 pro=1732681491.438292883,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680944.709853941,0 exp=1732681497.438292883,0 pro=1732681491.438292883,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680944.709853941,0 exp=1732681497.438292883,0 pro=1732681491.438292883,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680944.709853941,0 exp=1732681497.438292883,0 pro=1732681491.438292883,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.347707 295 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77665/1:‹/Table/106/1/-7439{363…-133…}›,raft] 7627  rejected lease request repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.451665447,0 pro=1732681491.451665447,0 acq=Request; current lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.451665447,0 pro=1732681491.451665447,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.451665447,0 pro=1732681491.451665447,0 acq=Request with repl=(n2,s2):3 seq=5 start=1732680898.860588054,0 exp=1732681497.451665447,0 pro=1732681491.451665447,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.350480 254 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77612/1:‹/Table/106/1/-7228{842…-612…}›,raft] 7628  rejected lease request repl=(n2,s2):3 seq=8 start=1732681005.992163752,0 exp=1732681497.446561273,0 pro=1732681491.446561273,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681005.992163752,0 exp=1732681497.446561273,0 pro=1732681491.446561273,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681005.992163752,0 exp=1732681497.446561273,0 pro=1732681491.446561273,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681005.992163752,0 exp=1732681497.446561273,0 pro=1732681491.446561273,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.352391 301 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77663/1:‹/Table/106/1/-741{9072…-8841…}›,raft] 7629  rejected lease request repl=(n2,s2):3 seq=6 start=1732681006.203415438,0 exp=1732681497.389024920,0 pro=1732681491.389024920,0 acq=Request; current lease repl=(n2,s2):3 seq=6 start=1732681006.203415438,0 exp=1732681497.389024920,0 pro=1732681491.389024920,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=6 start=1732681006.203415438,0 exp=1732681497.389024920,0 pro=1732681491.389024920,0 acq=Request with repl=(n2,s2):3 seq=6 start=1732681006.203415438,0 exp=1732681497.389024920,0 pro=1732681491.389024920,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.401832 257 kv/kvserver/queue.go:616 ⋮ [T1,Vsystem,n1,s1,r8012/1:‹/Table/106/1/-7155{748…-517…}›,raft] 7630  rate limited in MaybeAdd (replicate): throttled on async limiting semaphore
I241127 04:24:52.719430 275 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77624/1:‹/Table/106/1/-7244{522…-291…}›,raft] 7631  rejected lease request repl=(n2,s2):3 seq=8 start=1732681103.461766709,0 exp=1732681497.458348808,0 pro=1732681491.458348808,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681103.461766709,0 exp=1732681497.458348808,0 pro=1732681491.458348808,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681103.461766709,0 exp=1732681497.458348808,0 pro=1732681491.458348808,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681103.461766709,0 exp=1732681497.458348808,0 pro=1732681491.458348808,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.728405 278 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77652/1:‹/Table/106/1/-7412{385…-154…}›,raft] 7632  rejected lease request repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.442299075,0 pro=1732681491.442299075,0 acq=Request; current lease repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.442299075,0 pro=1732681491.442299075,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.442299075,0 pro=1732681491.442299075,0 acq=Request with repl=(n2,s2):3 seq=6 start=1732680914.192438268,0 exp=1732681497.442299075,0 pro=1732681491.442299075,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.740573 256 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77666/1:‹/Table/106/1/-7399{934…-703…}›,raft] 7633  rejected lease request repl=(n2,s2):3 seq=7 start=1732681006.185821758,0 exp=1732681497.500990996,0 pro=1732681491.500990996,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681006.185821758,0 exp=1732681497.500990996,0 pro=1732681491.500990996,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681006.185821758,0 exp=1732681497.500990996,0 pro=1732681491.500990996,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681006.185821758,0 exp=1732681497.500990996,0 pro=1732681491.500990996,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.891295 283 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77688/1:‹/Table/106/1/-124{3986…-1680…}›,raft] 7634  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.170839168,0 exp=1732681497.932766757,0 pro=1732681491.932766757,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.170839168,0 exp=1732681497.932766757,0 pro=1732681491.932766757,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.170839168,0 exp=1732681497.932766757,0 pro=1732681491.932766757,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.170839168,0 exp=1732681497.932766757,0 pro=1732681491.932766757,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.892386 290 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77669/1:‹/Table/106/1/-736{5116…-4885…}›,raft] 7635  rejected lease request repl=(n2,s2):3 seq=8 start=1732681006.258295597,0 exp=1732681497.952265470,0 pro=1732681491.952265470,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681006.258295597,0 exp=1732681497.952265470,0 pro=1732681491.952265470,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681006.258295597,0 exp=1732681497.952265470,0 pro=1732681491.952265470,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681006.258295597,0 exp=1732681497.952265470,0 pro=1732681491.952265470,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.893568 278 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77670/1:‹/Table/106/1/-7354{970…-740…}›,raft] 7636  rejected lease request repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.966478473,0 pro=1732681491.966478473,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.966478473,0 pro=1732681491.966478473,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.966478473,0 pro=1732681491.966478473,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732680914.218006748,0 exp=1732681497.966478473,0 pro=1732681491.966478473,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.910615 290 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77671/1:‹/Table/106/1/-115{4060…-1754…}›,raft] 7637  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.164930649,0 exp=1732681497.853065993,0 pro=1732681491.853065993,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.164930649,0 exp=1732681497.853065993,0 pro=1732681491.853065993,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.164930649,0 exp=1732681497.853065993,0 pro=1732681491.853065993,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.164930649,0 exp=1732681497.853065993,0 pro=1732681491.853065993,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.912038 297 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77693/1:‹/Table/106/1/-142{6146…-3840…}›,raft] 7638  rejected lease request repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681497.951162309,0 pro=1732681491.951162309,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681497.951162309,0 pro=1732681491.951162309,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681497.951162309,0 pro=1732681491.951162309,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681497.951162309,0 pro=1732681491.951162309,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.919906 291 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77685/1:‹/Table/106/1/-133{3913…-1607…}›,raft] 7639  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.163699103,0 exp=1732681497.888765201,0 pro=1732681491.888765201,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.163699103,0 exp=1732681497.888765201,0 pro=1732681491.888765201,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.163699103,0 exp=1732681497.888765201,0 pro=1732681491.888765201,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.163699103,0 exp=1732681497.888765201,0 pro=1732681491.888765201,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.930828 298 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77961/1:‹/Table/106/1/-7146{525…-294…}›,raft] 7640  rejected lease request repl=(n1,s1):1 seq=7 start=1732681492.535738489,0 exp=1732681498.896765920,0 pro=1732681492.896765920,0 acq=Request; current lease repl=(n1,s1):1 seq=7 start=1732681492.535738489,0 exp=1732681498.896765920,0 pro=1732681492.896765920,0 acq=Request; err: cannot replace lease repl=(n1,s1):1 seq=7 start=1732681492.535738489,0 exp=1732681498.896765920,0 pro=1732681492.896765920,0 acq=Request with repl=(n1,s1):1 seq=7 start=1732681492.535738489,0 exp=1732681498.896765920,0 pro=1732681492.896765920,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.930936 298 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77959/1:‹/Table/106/1/-7154{364…-134…}›,raft] 7641  rejected lease request repl=(n1,s1):1 seq=9 start=1732681492.537484035,0 exp=1732681498.896909686,0 pro=1732681492.896909686,0 acq=Request; current lease repl=(n1,s1):1 seq=9 start=1732681492.537484035,0 exp=1732681498.896909686,0 pro=1732681492.896909686,0 acq=Request; err: cannot replace lease repl=(n1,s1):1 seq=9 start=1732681492.537484035,0 exp=1732681498.896909686,0 pro=1732681492.896909686,0 acq=Request with repl=(n1,s1):1 seq=9 start=1732681492.537484035,0 exp=1732681498.896909686,0 pro=1732681492.896909686,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.976623 274 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77772/1:‹/Table/106/1/-{50151…-49920…}›,raft] 7642  rejected lease request repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.425097982,0 pro=1732681492.425097982,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.425097982,0 pro=1732681492.425097982,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.425097982,0 pro=1732681492.425097982,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.425097982,0 pro=1732681492.425097982,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:52.976778 282 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77768/1:‹/Table/106/1/-13{8002…-7772…}›,raft] 7643  rejected lease request repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.403773945,0 pro=1732681492.403773945,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.403773945,0 pro=1732681492.403773945,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.403773945,0 pro=1732681492.403773945,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.403773945,0 pro=1732681492.403773945,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.269636 256 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77824/1:‹/Table/106/1/-43{4645…-2340…}›,raft] 7644  rejected lease request repl=(n2,s2):3 seq=11 start=1732681104.428015188,0 exp=1732681498.456969973,0 pro=1732681492.456969973,0 acq=Request; current lease repl=(n2,s2):3 seq=11 start=1732681104.428015188,0 exp=1732681498.456969973,0 pro=1732681492.456969973,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=11 start=1732681104.428015188,0 exp=1732681498.456969973,0 pro=1732681492.456969973,0 acq=Request with repl=(n2,s2):3 seq=11 start=1732681104.428015188,0 exp=1732681498.456969973,0 pro=1732681492.456969973,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.297516 276 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77770/1:‹/Table/106/1/-1{72936…-49877…}›,raft] 7645  rejected lease request repl=(n2,s2):3 seq=12 start=1732681006.486438572,0 exp=1732681498.422052108,0 pro=1732681492.422052108,0 acq=Request; current lease repl=(n2,s2):3 seq=12 start=1732681006.486438572,0 exp=1732681498.422052108,0 pro=1732681492.422052108,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=12 start=1732681006.486438572,0 exp=1732681498.422052108,0 pro=1732681492.422052108,0 acq=Request with repl=(n2,s2):3 seq=12 start=1732681006.486438572,0 exp=1732681498.422052108,0 pro=1732681492.422052108,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.299529 281 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77692/1:‹/Table/106/1/-7{71294…-68989…}›,raft] 7646  rejected lease request repl=(n2,s2):3 seq=11 start=1732681104.219664381,0 exp=1732681498.476059685,0 pro=1732681492.476059685,0 acq=Request; current lease repl=(n2,s2):3 seq=11 start=1732681104.219664381,0 exp=1732681498.476059685,0 pro=1732681492.476059685,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=11 start=1732681104.219664381,0 exp=1732681498.476059685,0 pro=1732681492.476059685,0 acq=Request with repl=(n2,s2):3 seq=11 start=1732681104.219664381,0 exp=1732681498.476059685,0 pro=1732681492.476059685,0 acq=Request: ‹proposed under invalid lease›
W241127 04:24:53.303287 2292 2@rpc/clock_offset.go:286 ⋮ [T1,Vsystem,n1,rnode=2,raddr=‹10.142.0.100:26257›,class=raft,rpc] 7647  latency jump (prev avg 70.57ms, current 140.95ms)
I241127 04:24:53.349465 257 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77764/1:‹/Table/106/1/-56{8383…-6077…}›,raft] 7648  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.257212937,0 exp=1732681498.452588384,0 pro=1732681492.452588384,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.257212937,0 exp=1732681498.452588384,0 pro=1732681492.452588384,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.257212937,0 exp=1732681498.452588384,0 pro=1732681492.452588384,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.257212937,0 exp=1732681498.452588384,0 pro=1732681492.452588384,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.399496 301 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77691/1:‹/Table/106/1/-61{4499…-2193…}›,raft] 7649  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.271747314,0 exp=1732681498.405847987,0 pro=1732681492.405847987,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.271747314,0 exp=1732681498.405847987,0 pro=1732681492.405847987,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.271747314,0 exp=1732681498.405847987,0 pro=1732681492.405847987,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.271747314,0 exp=1732681498.405847987,0 pro=1732681492.405847987,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.400558 286 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77829/1:‹/Table/106/1/-705{3139…-2909…}›,raft] 7650  rejected lease request repl=(n2,s2):3 seq=7 start=1732681104.662127353,0 exp=1732681498.495160887,0 pro=1732681492.495160887,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681104.662127353,0 exp=1732681498.495160887,0 pro=1732681492.495160887,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681104.662127353,0 exp=1732681498.495160887,0 pro=1732681492.495160887,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681104.662127353,0 exp=1732681498.495160887,0 pro=1732681492.495160887,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.400772 291 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77765/1:‹/Table/106/1/-143{7675…-5369…}›,raft] 7651  rejected lease request repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.426114320,0 pro=1732681492.426114320,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.426114320,0 pro=1732681492.426114320,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.426114320,0 pro=1732681492.426114320,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.426114320,0 pro=1732681492.426114320,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.401041 301 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77689/1:‹/Table/106/1/-10{4107…-3876…}›,raft] 7652  rejected lease request repl=(n2,s2):3 seq=10 start=1732681104.203111988,0 exp=1732681498.400302351,0 pro=1732681492.400302351,0 acq=Request; current lease repl=(n2,s2):3 seq=10 start=1732681104.203111988,0 exp=1732681498.400302351,0 pro=1732681492.400302351,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=10 start=1732681104.203111988,0 exp=1732681498.400302351,0 pro=1732681492.400302351,0 acq=Request with repl=(n2,s2):3 seq=10 start=1732681104.203111988,0 exp=1732681498.400302351,0 pro=1732681492.400302351,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.402306 286 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77773/1:‹/Table/106/1/-83{8163…-5857…}›,raft] 7653  rejected lease request repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.513235948,0 pro=1732681492.513235948,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.513235948,0 pro=1732681492.513235948,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.513235948,0 pro=1732681492.513235948,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.513235948,0 pro=1732681492.513235948,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.405351 287 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77771/1:‹/Table/106/1/-5{91441…-89135…}›,raft] 7654  rejected lease request repl=(n2,s2):3 seq=10 start=1732681104.282733348,0 exp=1732681498.405898977,0 pro=1732681492.405898977,0 acq=Request; current lease repl=(n2,s2):3 seq=10 start=1732681104.282733348,0 exp=1732681498.405898977,0 pro=1732681492.405898977,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=10 start=1732681104.282733348,0 exp=1732681498.405898977,0 pro=1732681492.405898977,0 acq=Request with repl=(n2,s2):3 seq=10 start=1732681104.282733348,0 exp=1732681498.405898977,0 pro=1732681492.405898977,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.878694 295 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77893/1:‹/Table/106/1/-701{6016…-5785…}›,raft] 7655  rejected lease request repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681498.403138153,0 pro=1732681492.403138153,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681498.403138153,0 pro=1732681492.403138153,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681498.403138153,0 pro=1732681492.403138153,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681498.403138153,0 pro=1732681492.403138153,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.879365 295 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77687/1:‹/Table/106/1/-7352{664…-434…}›,raft] 7656  rejected lease request repl=(n2,s2):3 seq=10 start=1732681104.171312546,0 exp=1732681498.395464267,0 pro=1732681492.395464267,0 acq=Request; current lease repl=(n2,s2):3 seq=10 start=1732681104.171312546,0 exp=1732681498.395464267,0 pro=1732681492.395464267,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=10 start=1732681104.171312546,0 exp=1732681498.395464267,0 pro=1732681492.395464267,0 acq=Request with repl=(n2,s2):3 seq=10 start=1732681104.171312546,0 exp=1732681498.395464267,0 pro=1732681492.395464267,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.881076 281 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r78096/1:‹/Table/106/1/-8846{601…-371…}›,raft] 7657  rejected lease request repl=(n1,s1):1 seq=6 start=1732680946.251502356,0 exp=1732681499.646292340,0 pro=1732681493.646292340,0 acq=Request; current lease repl=(n1,s1):1 seq=6 start=1732680946.251502356,0 exp=1732681499.646292340,0 pro=1732681493.646292340,0 acq=Request; err: cannot replace lease repl=(n1,s1):1 seq=6 start=1732680946.251502356,0 exp=1732681499.646292340,0 pro=1732681493.646292340,0 acq=Request with repl=(n1,s1):1 seq=6 start=1732680946.251502356,0 exp=1732681499.646292340,0 pro=1732681493.646292340,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.881484 256 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77892/1:‹/Table/106/1/-7021{550…-319…}›,raft] 7658  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.737823411,0 exp=1732681498.828990376,0 pro=1732681492.828990376,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.737823411,0 exp=1732681498.828990376,0 pro=1732681492.828990376,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.737823411,0 exp=1732681498.828990376,0 pro=1732681492.828990376,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.737823411,0 exp=1732681498.828990376,0 pro=1732681492.828990376,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.884297 276 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77832/1:‹/Table/106/1/-51{3043…-0737…}›,raft] 7659  rejected lease request repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.815605265,0 pro=1732681492.815605265,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.815605265,0 pro=1732681492.815605265,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.815605265,0 pro=1732681492.815605265,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732680898.895548621,0 exp=1732681498.815605265,0 pro=1732681492.815605265,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.887287 254 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77830/1:‹/Table/106/1/-7031{695…-465…}›,raft] 7660  rejected lease request repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.862046868,0 pro=1732681492.862046868,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.862046868,0 pro=1732681492.862046868,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.862046868,0 pro=1732681492.862046868,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.862046868,0 pro=1732681492.862046868,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.887393 254 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77886/1:‹/Table/106/1/-8{61221…-58915…}›,raft] 7661  rejected lease request repl=(n2,s2):3 seq=11 start=1732681104.659832990,0 exp=1732681498.888766146,0 pro=1732681492.888766146,0 acq=Request; current lease repl=(n2,s2):3 seq=11 start=1732681104.659832990,0 exp=1732681498.888766146,0 pro=1732681492.888766146,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=11 start=1732681104.659832990,0 exp=1732681498.888766146,0 pro=1732681492.888766146,0 acq=Request with repl=(n2,s2):3 seq=11 start=1732681104.659832990,0 exp=1732681498.888766146,0 pro=1732681492.888766146,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.887427 274 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77856/1:‹/Table/106/1/-7030{542…-312…}›,raft] 7662  rejected lease request repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.867192644,0 pro=1732681492.867192644,0 acq=Request; current lease repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.867192644,0 pro=1732681492.867192644,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.867192644,0 pro=1732681492.867192644,0 acq=Request with repl=(n2,s2):3 seq=8 start=1732681006.618555831,0 exp=1732681498.867192644,0 pro=1732681492.867192644,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.887479 254 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77938/1:‹/Table/106/1/-696{2060…-1829…}›,raft] 7663  rejected lease request repl=(n2,s2):3 seq=7 start=1732680914.327070942,0 exp=1732681498.873809631,0 pro=1732681492.873809631,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680914.327070942,0 exp=1732681498.873809631,0 pro=1732681492.873809631,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680914.327070942,0 exp=1732681498.873809631,0 pro=1732681492.873809631,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680914.327070942,0 exp=1732681498.873809631,0 pro=1732681492.873809631,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.887594 274 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77862/1:‹/Table/106/1/-3{21661…-19355…}›,raft] 7664  rejected lease request repl=(n2,s2):3 seq=10 start=1732681104.665756358,0 exp=1732681498.927506220,0 pro=1732681492.927506220,0 acq=Request; current lease repl=(n2,s2):3 seq=10 start=1732681104.665756358,0 exp=1732681498.927506220,0 pro=1732681492.927506220,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=10 start=1732681104.665756358,0 exp=1732681498.927506220,0 pro=1732681492.927506220,0 acq=Request with repl=(n2,s2):3 seq=10 start=1732681104.665756358,0 exp=1732681498.927506220,0 pro=1732681492.927506220,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.888967 294 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77891/1:‹/Table/106/1/-7019{244…-013…}›,raft] 7665  rejected lease request repl=(n2,s2):3 seq=9 start=1732681104.774946571,0 exp=1732681498.939322250,0 pro=1732681492.939322250,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681104.774946571,0 exp=1732681498.939322250,0 pro=1732681492.939322250,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681104.774946571,0 exp=1732681498.939322250,0 pro=1732681492.939322250,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681104.774946571,0 exp=1732681498.939322250,0 pro=1732681492.939322250,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.888979 287 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77827/1:‹/Table/106/1/-84{9692…-7386…}›,raft] 7666  rejected lease request repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.970840235,0 pro=1732681492.970840235,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.970840235,0 pro=1732681492.970840235,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.970840235,0 pro=1732681492.970840235,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732680898.891379579,0 exp=1732681498.970840235,0 pro=1732681492.970840235,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:53.963316 282 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77914/1:‹/Table/106/1/-6966{441…-210…}›,raft] 7667  rejected lease request repl=(n2,s2):3 seq=7 start=1732680945.101768009,0 exp=1732681499.462366072,0 pro=1732681493.462366072,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680945.101768009,0 exp=1732681499.462366072,0 pro=1732681493.462366072,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680945.101768009,0 exp=1732681499.462366072,0 pro=1732681493.462366072,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680945.101768009,0 exp=1732681499.462366072,0 pro=1732681493.462366072,0 acq=Request: ‹proposed under invalid lease›
E241127 04:24:54.166709 2003938 (gostd) net/http/server.go:3416 ⋮ [-] 7668  ‹http: TLS handshake error from 10.0.0.3:54922: remote error: tls: bad certificate›
I241127 04:24:54.300452 288 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77859/1:‹/Table/106/1/-7055{445…-214…}›,raft] 7669  rejected lease request repl=(n2,s2):3 seq=7 start=1732681104.669675689,0 exp=1732681498.822003115,0 pro=1732681492.822003115,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681104.669675689,0 exp=1732681498.822003115,0 pro=1732681492.822003115,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681104.669675689,0 exp=1732681498.822003115,0 pro=1732681492.822003115,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681104.669675689,0 exp=1732681498.822003115,0 pro=1732681492.822003115,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.301285 291 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77977/1:‹/Table/106/1/-7157{823…-593…}›,raft] 7670  rejected lease request repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.301393 291 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77977/1:‹/Table/106/1/-7157{823…-593…}›,raft] 7671  rejected lease request repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681492.924292743,0 exp=1732681499.151797778,0 pro=1732681493.151797778,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.302119 299 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77917/1:‹/Table/106/1/-701{8091…-7860…}›,raft] 7672  rejected lease request repl=(n2,s2):3 seq=9 start=1732681006.632826521,0 exp=1732681499.482218238,0 pro=1732681493.482218238,0 acq=Request; current lease repl=(n2,s2):3 seq=9 start=1732681006.632826521,0 exp=1732681499.482218238,0 pro=1732681493.482218238,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=9 start=1732681006.632826521,0 exp=1732681499.482218238,0 pro=1732681493.482218238,0 acq=Request with repl=(n2,s2):3 seq=9 start=1732681006.632826521,0 exp=1732681499.482218238,0 pro=1732681493.482218238,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.302551 290 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77885/1:‹/Table/106/1/-37{7000…-4694…}›,raft] 7673  rejected lease request repl=(n2,s2):3 seq=11 start=1732681006.600382265,0 exp=1732681498.945015993,0 pro=1732681492.945015993,0 acq=Request; current lease repl=(n2,s2):3 seq=11 start=1732681006.600382265,0 exp=1732681498.945015993,0 pro=1732681492.945015993,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=11 start=1732681006.600382265,0 exp=1732681498.945015993,0 pro=1732681492.945015993,0 acq=Request with repl=(n2,s2):3 seq=11 start=1732681006.600382265,0 exp=1732681498.945015993,0 pro=1732681492.945015993,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.302619 289 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77767/1:‹/Table/106/1/-13{9155…-8925…}›,raft] 7674  rejected lease request repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.983237201,0 pro=1732681492.983237201,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.983237201,0 pro=1732681492.983237201,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.983237201,0 pro=1732681492.983237201,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680885.432453374,0 exp=1732681498.983237201,0 pro=1732681492.983237201,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.305486 284 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77854/1:‹/Table/106/1/-703{7229…-6999…}›,raft] 7675  rejected lease request repl=(n2,s2):3 seq=7 start=1732681104.670196601,0 exp=1732681498.849499885,0 pro=1732681492.849499885,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732681104.670196601,0 exp=1732681498.849499885,0 pro=1732681492.849499885,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732681104.670196601,0 exp=1732681498.849499885,0 pro=1732681492.849499885,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732681104.670196601,0 exp=1732681498.849499885,0 pro=1732681492.849499885,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.306073 277 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77942/1:‹/Table/106/1/-6939{463…-232…}›,raft] 7676  rejected lease request repl=(n2,s2):3 seq=7 start=1732680914.334657653,0 exp=1732681498.910006754,0 pro=1732681492.910006754,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680914.334657653,0 exp=1732681498.910006754,0 pro=1732681492.910006754,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680914.334657653,0 exp=1732681498.910006754,0 pro=1732681492.910006754,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680914.334657653,0 exp=1732681498.910006754,0 pro=1732681492.910006754,0 acq=Request: ‹proposed under invalid lease›
I241127 04:24:54.307559 275 kv/kvserver/kvserverbase/forced_error.go:87 ⋮ [T1,Vsystem,n1,s1,r77934/1:‹/Table/106/1/-700{7023…-6792…}›,raft] 7677  rejected lease request repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681499.434201063,0 pro=1732681493.434201063,0 acq=Request; current lease repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681499.434201063,0 pro=1732681493.434201063,0 acq=Request; err: cannot replace lease repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681499.434201063,0 pro=1732681493.434201063,0 acq=Request with repl=(n2,s2):3 seq=7 start=1732680891.229233672,0 exp=1732681499.434201063,0 pro=1732681493.434201063,0 acq=Request: ‹proposed under invalid lease›
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678  a panic has occurred!
E241127 04:24:54.308144 301 1@util/log/logcrash/crash_reporting.go:192 â‹® [T1,Vsystem,n1] 7678 +pull mode must not have sending entries

• test artifacts

Jira issue: CRDB-44973