roachtest: create test where node rejoins cluster after being down for multiple hours

[kvoli]

Is your feature request related to a problem? Please describe.

When a cockroach node rejoins the cluster, its (remaining) replicas need to be caught up. The catchup can cause IO saturation leading to increased tail latency.

Describe the solution you'd like

Write a roachtest, Similar to kv/restart/nodes=12, which brings nodes down for 10 minutes. In this test, a node should remain down for at least an hour and contain a large enough number of replicas (bytes) that it will not have all its replicas replaced within the hour.

The test should assert or observe the impact of the rejoining node on cluster throughput, where a moderately write heavy workload will run throughout.

See earlier work done in #96521 and catch up pacing proposed in #98710.

Jira issue: CRDB-34142

[andrewbaptist]

Some notes about recreating this manually
Create cluster:

roachprod create -n 13 $CLUSTER
roachprod stage $CLUSTER release v23.1.12 
roachprod start $CLUSTER:1-12\

Install docker

roachprod ssh $CLUSTER:13
sudo apt  install docker.io

Init tpc-e - takes ~1 hour

roachprod ssh $CLUSTER:13 "sudo docker run cockroachdb/tpc-e:latest --init -c25000 --hosts $(roachprod ip $CLUSTER:1)"

Run test against first 11 nodes

roachprod ssh $CLUSTER:13 "sudo docker run cockroachdb/tpc-e:latest -c25000 -d30m $(roachprod ip $CLUSTER:1-11 | sed s/^/--hosts=/ | tr '\n' ' ')"

• Wait 5 minutes for the workload,
• Gracefully stop node 12
• Wait 5 more minutes
• Start node 12

sleep 300
roachprod stop $CLUSTER:12 --sig 15
sleep 300
roachprod start $CLUSTER:12

[andrewbaptist]

I have some good manual reproduction of this and will turn into a roachtest tomorrow.
The set of commands is:

roachprod create -n 13 $CLUSTER --local-ssd
roachprod stage $CLUSTER release v23.1.12 
roachprod start $CLUSTER:1-12

roachprod ssh $CLUSTER:1 "./cockroach workload init kv --splits 5000"
roachprod ssh $CLUSTER:13 "./cockroach workload run kv {pgurl:1-11} --read-percent 10 --max-block-bytes 4096 --concurrency 256 --max-rate 12000"

# Wait for ~2 hours to let sufficient fill in the system. The workload continues during the following steps.

roachprod stop $CLUSTER:12 --sig 15
sleep 600
roachprod start $CLUSTER:12

22.2.13 default settings:

23.1.13 default settings:

23.1.13 with a change to three setting:

• AC disabled for foreground traffic (default is on for foreground and background)
• Lease transfer enforcement set to shed (default is block_transfer_to)
• Lease IO overload set to 0.3 (default is 0.5 in 23.1 and 0.3 in 23.2)

[andrewbaptist]

As part of this investigation, I've found a number of reasons for slowdowns after the node is shut down and restarted

1. Raft catchup traffic floods the disk
2. High scan rate against meta2 during lease and replica movement (both after the node is dead and after it comes back)
3. High snapshot transfer rate can slow down foreground traffic
4. Deletion for replicas that are moved off the node that is down cause compaction to focus on lower layers
5. Premature movement of leases to the node that is overloaded
6. AC slowing foreground traffic due to high goroutines and IO overload

The changes to the test in #116291 expose many of these issues. We are still far away from having no impact, but this test can be used as a benchmark to improve the situation over time.