kv: replicate unreplicated locks before lease transfers, splits, and merges

[nvanbenschoten]

Currently, a cooperative lease transfer discards the entire lock table on the outgoing leaseholder and rebuilds the lock table on the incoming leaseholder lazily. This means that any unreplicated lock on the outgoing leaseholder will be lost. This causes the best-effort mutual exclusion provided by unreplicated locks (those acquired using implicit or explicit SELECT ... FOR UPDATE) to be less reliable. This can in turn lead to transaction retries and other surprising behavior.

To avoid this, we could ship the unreplicated locks from the outgoing leaseholder to the incoming leaseholder through the lease transfer Raft proposal. This would be similar to #60521, where we started including more information in the handoff between leaseholders during cooperative lease changes.

If we make this change, we will also want to consider the corresponding situations with range splits and merges.

---

EDIT: in #100193, we added support for replicated locks. Instead of shipping unreplicated locks between leaseholders using a new mechanism, it probably makes sense to instead flush unreplicated locks to the replicated state machine (push them through raft) ahead of performing a lease transfer, a split, or a merge.

Jira issue: CRDB-12681

Epic CRDB-23101

[nvanbenschoten]

We have no near-term plans to ship unreplicated key-level locks between leaseholders. Instead, we implemented replicated key-level locks (#100193) in v23.2. This new form of lock is replicated through Raft immediately upon acquisition, similar to other writes. Consequently, they are not lost during lease transfers, range splits, or range merges. Replicated locks were added for READ COMMITTED isolation, but can also be used in SERIALIZABLE isolation.

To do this, users who are running v23.2 can configure the enable_durable_locking_for_serializable session variable in any of the various ways that session variables are configured. For example, a transaction that wants replicated FOR UPDATE locks can be written like:

BEGIN;
SET LOCAL enable_durable_locking_for_serializable = true;
SELECT ... FOR UPDATE;
...
COMMIT;

There is a latency cost to this, which is proportional to the raft replication round-trip time. We are hoping to hide this latency using a pipelining technique in v24.1 (#117978).

[nvanbenschoten]

Now that we support replicated key-level locks (#100193), the natural way to implementing this would be to "flush" unreplicated locks to the replicated state machine before lease transfers, splits, and merges. I'll repurpose this issue for that different approach for solving the same problem.