kvserver: proactively enqueue replicas for a decommissioning node #79453
Assignees
Labels
A-kv-distribution
Relating to rebalancing and leasing.
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
O-postmortem
Originated from a Postmortem action item.
T-kv
KV Team
Comments
aayushshah15
added
C-enhancement
|
Related to #79318. |
This was referenced May 2, 2022
|
manually reviewed and brought up to date |
exalate-issue-sync
bot
removed
the
A-kv-decom-rolling-restart
craig bot
pushed a commit
that referenced
this issue
Jun 12, 2022
81005: kvserver: retry failures to rebalance decommissioning replicas r=aayushshah15 a=aayushshah15 Related to #80993 Relates to #79453 This commit makes it such that failures to rebalance replicas on decommissioning nodes no longer move the replica out of the replicateQueue as they previously used to. Instead, these failures now put these replicas into the replicateQueue's purgatory, which will retry these replicas every minute. All this is intended to improve the speed of decommissioning towards its tail end, since previously, failures to rebalance these replicas meant that they were only retried after about 10 minutes. Release note: None Co-authored-by: Aayush Shah <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Came up in a conversation with, and proposed by, @lidorcarmel.
Today, the
replicaScanneron each store in the cluster keeps pacing through its replicas (such that it processes each replica once roughly every 10 mins). ThereplicaScannerkeeps optionally queuing these replicas into each of the store's queues.When a node is decommissioning, this status gets broadcast to other nodes in the cluster via gossip. So when the replicateQueue encounters a replica that is the leaseholder for a range that has a replica on a decommissioning node, it decides to take action to move that decommissioning replica away.
The issue here is that this discovery of decommissioning replicas is limited by the
replicaScanners 10 min scanning interval. This means that, generally, even the discovery of all replicas belonging to a decommissioning node will take ~10 mins. Furthermore, if there are any errors processing any of these decommissioning replicas, they will not be re-processed for another 10 mins.This issue proposes that we should actively enqueue all replicas belonging to a decommissioning node, into the
replicateQueues of all of a node's stores the moment it learns that a node's liveness record has changed fromLIVEtoDECOMMISSIONING. Care will need to be taken to ensure that we're only enqueuing these replicas exactly once when the status of a node changes toDECOMMISSIONING. Doing this should considerably cut down on how long it takes to decommission nodes in almost all scenarios, and, anecdotally, it also seems like the behaviour that operators intuitively expect.cc @cockroachdb/kv-notifications
Jira issue: CRDB-14873
Epic: CRDB-14621
The text was updated successfully, but these errors were encountered: