Shutdown QQ peer members on force-shrink-to-single-member execution to allow restoring QQ clusters on healthy nodes

[Ayanda-D]

Proposed Changes

Hi

This is a follow up to #12344 (bouncing off part of our approach to some DR scenarios in case of any unforeseen issues).

We rely on force-shrink commands to rescue QQ nodes in extreme disaster scenarios. Current behavior, when QQs are shrunk to a single node, connected peers keep their local FSMs running, which prevents re-growing the shrunk queues back onto connected nodes which would/could be in a healthy state. We expect peers to update their log to that of the leader on grow and/or the next received #append_entries_rpc{}.

Example, doing force_all_queues_shrink_member_to_current_member/0.

$> rabbitmq-queues quorum_status Q1
Status of quorum queue Q1 on node rabbit@host ...
┌─────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name           │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit_2@host │ follower   │ 233       │ 233          │ undefined      │ 16   │ 3               │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit_1@host │ follower   │ 233       │ 233          │ undefined      │ 16   │ 3               │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@host   │ leader     │ 233       │ 233          │ undefined      │ 16   │ 3               │
└─────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘


$> rabbitmqctl eval 'rabbit_quorum_queue:force_all_queues_shrink_member_to_current_member().'
ok


$> rabbitmq-queues quorum_status Q1
Status of quorum queue Q1 on node rabbit@host ...
┌───────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name         │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@host │ leader     │ 235       │ 235          │ undefined      │ 17   │ 3               │
└───────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

At this point, the cluster can't be restored. i.e. queues cant regrow back to the rest of the cluster (even if they're healthy).

With this patch, if after disaster scenario, part of the cluster nodes are reachable and healthy, we are able to proceed with the re-growing the QQs back to healthy nodes without undergoing major recovery procedures, as follows:

$> rabbitmq-queues grow 'rabbit_1@host' all --queue-pattern Q1
Growing all quorum queues on rabbit_1@host...
vhost	name	size	result
/	Q1	2	ok

$> rabbitmq-queues grow 'rabbit_2@host' all --queue-pattern Q1
Growing all quorum queues on rabbit_2@host...
vhost	name	size	result
/	Q1	3	ok


$> rabbitmq-queues quorum_status Q1
Status of quorum queue Q1 on node rabbit@host ...
┌─────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name           │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit_2@host │ follower   │ 237       │ 237          │ undefined      │ 17   │ 3               │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit_1@host │ follower   │ 237       │ 237          │ undefined      │ 17   │ 3               │
├─────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@host   │ leader     │ 237       │ 237          │ undefined      │ 17   │ 3               │
└─────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

This allows us to avoid unnecessarily doing major recovery procedures like complete wipe out of peers node and/or migrations and queue deletions/re-creation, etc, to restore the cluster. At this point of the cluster, the only other option is a complete re-deploy.

As part of this PR/proposal, we also introduce force_vhost_queues_shrink_member_to_current_member/1 which allows us to force-shink all QQs on a per-vhost basis (which we need/use for some our shared cluster environments).

Please take a look 😬

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• [] I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

[michaelklishin]

Per discussion with @kjnilsson: the change makes sense. The function that stops all replicas in a virtual host was introduced to be used by tests.

[michaelklishin]

#12475 is merged, #12468 needs some fixes (on Erlang 27) not related to this PR.