kvserver: v23.1.2-custom: attempted to change replica's ID from 33 to 20

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/4249345938/?referrer=webhooks_plugin

Panic message:

replica_init.go:361: log.Fatal: attempted to change replica's ID from 33 to 20
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:361
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setDescRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:329
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleDescResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go:318
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:331
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:210
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1005
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) log.Fatal: attempted to change replica's ID from 33 to 20
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: attempted to change replica's ID from 33 to 20
replica_init.go:361: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L360-L362 in pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L328-L330 in pkg/kv/kvserver.(*Replica).setDescRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go#L317-L319 in pkg/kv/kvserver.(*Replica).handleDescResult
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L330-L332 in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L186-L188 in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/apply/cmd.go#L209-L211 in pkg/kv/kvserver/apply.mapCheckedCmdIter
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/apply/task.go#L294-L296 in pkg/kv/kvserver/apply.(*Task).applyOneBatch
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/apply/task.go#L250-L252 in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1004-L1006 in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L717-L719 in pkg/kv/kvserver.(*Replica).handleRaftReady
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647 in pkg/kv/kvserver.(*Store).processReady
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396 in pkg/kv/kvserver.(*raftSchedulerShard).worker
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303 in pkg/kv/kvserver.(*raftScheduler).Start.func2
https://github.com/cockroachdb/cockroach/blob/7f0603d07117c140efb99c254bfa61d715f776ca/pkg/util/stop/stopper.go#L469-L471 in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked at line 361
pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).setDescRaftMuLocked at line 329
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go in pkg/kv/kvserver.(*Replica).handleDescResult at line 318
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult at line 331
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects at line 187
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCheckedCmdIter at line 210
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 295
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1005
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 718
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594

Tag	Value
Cockroach Release	v23.1.2-2-g7f0603d0711
Cockroach SHA:	7f0603d07117c140efb99c254bfa61d715f776ca
Platform	linux amd64
Distribution	CCL
Environment	v23.1.2
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-28744

[blathers-crl]

cc @cockroachdb/replication

[tbg]

This is this internal testing POC cluster.

[tbg]

This is similar to #94813, which I investigated but never understood. I had then sent a WIP #102248 but never pushed it over the line. It might be time to do so now.

[erikgrinaker]

This came from an internal cluster that was running without fsync.