kvserver: a slow follower should not stall the quota pool

[erikgrinaker]

The quota pool will halt proposals when a certain amount of proposals are in flight and not yet replicated and applied on all replicas. This means that a single slow follower can stall the entire range. There are exceptions here, i.e. we ignore followers we haven't communicated with in a while, and also pause followers that are IO overloaded:

cockroach/pkg/kv/kvserver/replica_proposal_quota.go

Lines 172 to 174 in d80a9f5

	if !r.mu.lastUpdateTimes.isFollowerActiveSince(rep.ReplicaID, now, r.store.cfg.RangeLeaseDuration) {
	return
	}

cockroach/pkg/kv/kvserver/replica_proposal_quota.go

Lines 217 to 224 in d80a9f5

	if _, paused := r.mu.pausedFollowers[roachpb.ReplicaID(id)]; paused {
	// We are dropping MsgApp to this store, so we are effectively treating
	// it as non-live for the purpose of replication and are letting it fall
	// behind intentionally.
	//
	// See #79215.
	return
	}

However, if the follower is live but simply slow, then it will drag the whole range down with it. We saw this happen e.g. in #113053, where the follower was stuck in an append loop, and kept rejecting MsgApps for hours, which stalled the range.

We shouldn't let a minority of followers stall the range. Instead, we should give up on them and allow progress with the remaining quorum. However, this requires the allocator to upreplicate elsewhere, since the range will otherwise stall if we lose one of the healthy replicas.

This is related to several previous discussions, and also replication admission control which we hope will replace the quota pool in the longer term. For example:

• kvserver: throttle writes on followers #79215
• admission,kv,bulk: unify (local) store overload protection via admission control #75066
• kvserver: ignore draining nodes in proposal quota #55806
• kvserver: disable & delete the quota pool #106063

Jira issue: CRDB-32733

Epic CRDB-39900

[blathers-crl]

cc @cockroachdb/replication

[mzone242]

Hello, @annabelto and I would like to take this issue. We are Masters students at The University of Texas at Austin and are looking to contribute to an open source project concerned with distributed systems for our coursework in the subject.

Can you also provide some more information on the deliverables of this issue? Specifically, we're unfamiliar with "upreplication". We can see that the long term plan is to phase out this feature, but would like to help address it in the short term.

[erikgrinaker]

Hi @mzone242, thanks for your interest! We haven't decided on a direction here yet, but may know more next week.

[erikgrinaker]

We're leaning towards removing the quota pool outright, since Raft will naturally pace replication at the quorum rate. See #106063.

[mzone242]

Is that something we can help with at this time?

[erikgrinaker]

We can start by adding a cluster setting that allows disabling the quota pool. I wrote up a brief outline over in #106063 (comment), let's continue the discussion on that issue.