kvserver: don't quiesce with follower in StateProbe

[tbg]

shouldReplicaQuiesce checks that all followers are fully caught up, but it's
still possible to have a follower in StateProbe (because we call
rawNode.ReportUnreachable when an outgoing message gets dropped).

Persistent StateProbe is problematic: we check for it before lease transfers, so
despite the follower being fully caught up, we'd refuse the transfer.

Unfortunately, this commit itself is not enough: even if the range is not
quiesced, it needs to replicate a new entry to rectify the situation (i.e.
switch follower back to StateReplicate). This is because at the time of writing,
receipt of a heartbeat response from the follower is not enough to move it back
to StateReplicate.
This was fixed upstream, in #103826.

However, this is still not enough! If the range quiesces successfully, and
then ReportUnreachable is called, we still end up in the same state; this is now tracked in #103828.

I ran into the above issue on
#99191, which adds persistent
circuit breakers, when stressing TestStoreMetrics. That test happens to
restart n2 when it's fully caught up and due to the persistence of the circuit
breakers when it comes up the leader will move it into StateProbe (since we can
end up dropping the first heartbeat sent to it as it comes up, since the breaker
hasn't untripped yet).

But, I believe that this bug is real even without this breaker re-work, just
harder to trigger.

Epic: none
Release note (bug fix): fixed a problem that could lead to erroneously refused
lease transfers (error message: "refusing to transfer lease to [...] because
target may need a Raft snapshot: replica in StateProbe"

[cockroach-teamcity]

This change is 

[erikgrinaker]

LGTM, thanks! Backportable?

[tbg]

TFTR!
bors r=erikgrinaker

Backports should be fine, I'll give it a try.

Test failures in acceptance/version-upgrade are unrelated:

Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: index element ?c😀ol6_9 of type jsonb is not indexable in a non-inverted index (SQLSTATE 42P16)

Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: index element "c%vol9"_10 of type jsonb is not indexable in a non-inverted index (SQLSTATE 42P16)

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[tbg]

I am stumped by how I've gotten this mixed version error three times in a row. The test has been flaky for a while, but I seem to be getting exceptionally unlucky (slack thread¹).
Going to run it a few more times locally just to check there can't possibly be anything in this PR triggering this (not sure how that would even be possible).

Footnotes

1. https://cockroachlabs.slack.com/archives/C016CAD2HQ8/p1684931703555999 ↩

[tbg]

bors r=erikgrinaker

Passes on gceworker.

[craig]

This PR was included in a batch that was canceled, it will be automatically retried

[tbg]

bors r=erikgrinaker

[craig]

Already running a review

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 0e608b0 to blathers/backport-release-22.2-103827: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}