kvserver: fix 'observed raft log position' assertion

Fixes #107336.
Fixes #106123.
Fixes #107156.
Fixes #106589.

It's possible to hit this assertion under --stress --race when the
proposing replica is starved enough for raft ticks that it loses
leadership right when it steps proposals through raft. We're relying on
undocumented API semantics in the etcd raft library whereby it mutates
stepped entries with the term+index its to end up in. But that's only
applicable if stepping through entries as a leader. Simply relax this
assertion instead.

Release note: None

pkg/kv/kvserver/replica_proposal_buf.go

@@ -1043,6 +1043,16 @@ func maybeDeductFlowTokens(
 		if admitHandle.handle == nil {
 			continue // nothing to do
 		}
+		if ents[i].Term == 0 && ents[i].Index == 0 {
+			// It's possible to have lost raft leadership right before stepping
+			// proposals through raft. They'll get forwarded to the new raft
+			// leader, and for flow token purposes, there's no tracking
+			// necessary. The token deductions below asserts on monotonic
+			// observations of log positions, which this empty position would
+			// otherwise violate. There's integration code elsewhere that will
+			// free up all tracked tokens as a result of this leadership change.
+			return
+		}
 		log.VInfof(ctx, 1, "bound index/log terms for proposal entry: %s",
 			raft.DescribeEntry(ents[i], func(bytes []byte) string {
 				return "<omitted>"