kvserver: replicaGCQueue acquires latch reaching into another rangeID

[tbg]

Describe the problem

@irfansharif reported this log from a WIP test¹. In the test, r64 lost quorum and the circuit breaker tripped. But then we see this on r1, which should be healthy (it's a single-voter group):

E230507 19:48:05.815591 17777 kv/kvserver/queue.go:1136 [T1,n2,replicaGC,s2,r64/2:/{Table/Max-Max}] 599 replica unavailable: (n1,s1):1 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, next=2, gen=0]: closed timestamp: 1683488882.708838000,0 (2023-05-07 19:48:02); raft status: {"id":"1","term":6,"vote":"1","commit":51,"lead":"1","raftState":"StateLeader","applied":51,"progress":{"1":{"match":51,"next":52,"state":"StateReplicate"}},"leadtransferee":"0"}: encountered poisoned latch /Local/RangeID/64/r/AbortSpan/"8dfa719f-0a1a-40fc-a56f-a174787edc56"@0,0

and

E230507 19:48:05.815646 653 kv/kvserver/reports/reporter.go:159 [T1,n1,replication-reporter] 598 failed to generate replication reports: failed to compute constraint conformance report: replica unavailable: (n1,s1):1 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, next=2, gen=0]: closed timestamp: 1683488882.708838000,0 (2023-05-07 19:48:02); raft status: {"id":"1","term":6,"vote":"1","commit":51,"lead":"1","raftState":"StateLeader","applied":51,"progress":{"1":{"match":51,"next":52,"state":"StateReplicate"}},"leadtransferee":"0"}: encountered poisoned latch /Local/RangeID/64/r/AbortSpan/"8dfa719f-0a1a-40fc-a56f-a174787edc56"@0,0

Almost certainly this will have something to do with the fact that the first range has a start key of KeyMin (empty key) but the addressable keyspace only starts at keys.LocalMax (\x02). Both replicaGC and constraint reports scan meta2, which is contained in r1 in this test. In doing so they seem to acquire a latch that spans range-local replicated keys for r64 (and likely all other ranges). This is incorrect.

I don't understand how that latch gets into r1's latch manager. Each replica has its own latch manager so even if r64 has a bunch of poisoned latches why does it show up in r1?

Jira issue: CRDB-27742

Footnotes

1. https://cockroachlabs.slack.com/archives/G01G8LK77DK/p1683495028679899 ↩

[irfansharif]

To reproduce, I used the following from #98308. One oddity was that I was suppressing time-based election timeouts, so I'd use RaftElectionTimeoutTicks: 10000.

$ dev test pkg/kv/kvserver -f TestFlowControlRaftMembershipRemoveSelf/transfer-lease-first=true \
  -v --show-logs --stream-output --timeout 10m --stress --ignore-cache