kvserver: lease transferred to uninitialized follower

[tbg]

Describe the problem

In #61541 (comment), we saw that a range with an uninitialized follower transferred the lease to that uninitialized follower, leading to the range becoming unavailable until the follower caught up. This indicates a hole in our safety mechanisms.

To Reproduce

Unclear, but the test in #61541 definitely achieved it.

Expected behavior
A clear and concise description of what you expected to happen.

Additional data / screenshots

Environment:

https://github.com/cockroachdb/cockroach/commits/b703e663da8ededaee2e28fc39a24e3880ae54cf

Additional context

It looks like aggressive changes in range size can lead to temporary (but substantial) unavailability.

gz#9425

[tbg]

@nvanbenschoten are you aware of anything having changed around lease transfers? We need to know whether to classify #61541 as a release blocker.

[nvanbenschoten]

A decent amount has changed around lease transfers, but I don't think it would have affected this.

As @aayushshah15 pointed out in #61541:

we don’t seem to do anything to prevent lease transfers to VOTER_FULL replicas that are waiting for a snapshot

One thing to point out is that we recently made splits/largerange/size=32GiB,nodes=6 more aggressive. We bumped the range size up from 10GB to 32GB and bumped the cluster size up from 3 nodes to 6 nodes. Aayush is aware of this and is looking to determine whether either of those changes is the cause of the newer failures. We'll want to make the test less aggressive again if we determine that there's nothing immediately actionable in how we respond to this much splitting/rebalancing.

[aayushshah15]

@andreimatei and I were quibbling about this and it seems like a worthwhile safeguard we can add on a leader is just to reject TransferLeaseRequests to followers that its progress-tracker knows are in StateSnapshot (though I'm not sure at what "level" we want to reject such a thing, maybe during the command evaluation?).

[aayushshah15]

Looks like we had a much more aggressive version of this sort of safeguard that we removed entirely in this patch: #42379