sentry: store_raft.go:524: log.Fatal: ???: raft closed timestamp regression in cmd: "^\xb1\x9f\xfa3HO\xb2" (term: 42, index: 3279670); batch state: 1652071361.888790000,0, command: 1651969445.464306000,0, lea...

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3261796813/?referrer=webhooks_plugin

Panic message:

store_raft.go:524: log.Fatal: ???: raft closed timestamp regression in cmd: "^\xb1\x9f\xfa3HO\xb2" (term: 42, index: 3279670); batch state: 1652071361.888790000,0, command: 1651969445.464306000,0, lease: repl=(n1,s1):1 seq=36 start=1651206959.965603000,0 exp=1652072838.515903000,0 pro=1652072829.515903000,0, req: <unknown; not leaseholder>, applying at LAI: 65885.
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:1099
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:465
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:175
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:280
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:247
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:803
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:466
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:523
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:284
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1374
Wraps: (3) raft closed timestamp regression in cmd: "^\xb1\x9f\xfa3HO\xb2" (term: 42, index: 3279670); batch state: 1652071361.888790000,0, command: 1651969445.464306000,0, lease: repl=(n1,s1):1 seq=36 start=1651206959.965603000,0 exp=1652072838.515903000,0 pro=1652072829.515903000,0, req: <unknown; not leaseholder>, applying at LAI: 65885.
| Closed timestamp was set by req: <unknown; not leaseholder or not lease request> under lease: %!s(PANIC=SafeFormatter method: value method github.com/cockroachdb/cockroach/pkg/roachpb.Lease.SafeFormat called using nil *Lease pointer); applied at LAI: 0. Batch idx: 0.
| This assertion will fire again on restart; to ignore run with env var COCKROACH_RAFT_CLOSEDTS_ASSERTIONS_ENABLED=trueRaft log tail:
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
--
*errutil.leafError: log.Fatal: ???: raft closed timestamp regression in cmd: "^\xb1\x9f\xfa3HO\xb2" (term: 42, index: 3279670); batch state: 1652071361.888790000,0, command: 1651969445.464306000,0, lease: repl=(n1,s1):1 seq=36 start=1651206959.965603000,0 exp=1652072838.515903000,0 pro=1652072829.515903000,0, req: <unknown; not leaseholder>, applying at LAI: 65885. (1)
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack::: (2)
store_raft.go:524: *withstack.withStack (top exception)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvserver/store_raft.go

Lines 523 to 525 in 1fd13c9

	stats, expl, err := r.handleRaftReady(ctx, noSnap)
	removed := maybeFatalOnRaftReadyErr(ctx, expl, err)
	elapsed := timeutil.Since(start)

in pkg/kv/kvserver.(*Store).processReady

cockroach/pkg/kv/kvserver/scheduler.go

Lines 283 to 285 in 1fd13c9

	if state.flags&stateRaftReady != 0 {
	s.processor.processReady(ctx, id)
	}

in pkg/kv/kvserver.(*raftScheduler).worker

cockroach/pkg/util/stop/stopper.go

Lines 350 to 352 in 1fd13c9

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTask.func1
/usr/local/go/src/runtime/asm_amd64.s#L1373-L1375 in runtime.goexit

pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 524
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).worker at line 284
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTask.func1 at line 351
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1374

Tag	Value
Cockroach Release	v21.1.5
Cockroach SHA:	1fd13c9
Platform	darwin amd64
Distribution	CCL
Environment	development
Command	start-single-node
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-15321

[yuzefovich]

dup of #70894