Make Raft trailing logs and snapshot timing reloadable

[banks]

This is a partial fix for #9609.

The description in that issue gives lots of background although much of it is also present in the telemetry docs proposed in this PR too.

This PR:

• Updates Raft and go-metrics to pull in new telemetry and hot reloadable features
• Makes the raft options reloadable in Consul
• Updates our pre-registered prometheus metrics to cover the new raft ones and the old ones I've not added to the telemetry guide as important
• Updates telemetry and config docs with the changes in metrics/reload behaviour
  • Sorry, but I also re-organised the raft metrics to be alphabetical - I've made the mistake of thinking we missed metrics on two separate occasions because they were not grouped alphabetically and even not hierarchically (e.g. some raft.fsm metrics are not alongside others... Probably makes the diff more onerous than it should be.)
• Adds a "Key Metric" section to the telemetry docs that describes this issue in more detail and how to use the new metrics to monitor for it.

Feedback Requested

The Key Metrics section is verbose but it's pretty hard to simplify describing how to deal with this case without that detail. I feel like it would be better suited for more of a "operator runbook" section where we can go into a bit more detail on debugging common or serious failure modes of Consul, but we have no such thing yet and I wanted to get this available at least before taking on the project of starting a whole new type of content in the docs.

Would love feedback on whether it feels sufficient/appropriate/clear enough or if there are other ideas to improve. I'd probably defer any major changes to docs structure to later rather than take them on in this PR though as I want to get this into the next 1.10 beta.

Testing

The reloadable stuff has tests here but I also spent a while trying this out for real to make sure the metrics actually work and are useful. For example:

Here the leader has an artificially low raft_trailing_logs of 1 initially. It is writing our 1GB snapshots roughly every 2 minutes and is handling about 200 writes a second.

At 12:01 I changed raft_trailing_logs to 100k and used consul_reload the same leader (s1) continued to snapshot another couple of times but without truncating logs.

I also tested changes to snapshot threshold and timing.

TODO

• Add changelogs for these changes, but also check for other fixes in Raft 1.3.0 that could impact Consul users and add those too

[hashicorp-ci]

🤔 This PR has changes in the website/ directory but does not have a type/docs-cherrypick label. If the changes are for the next version, this can be ignored. If they are updates to current docs, attach the label to auto cherrypick to the stable-website branch after merging.

[mkeeler]

LGTM, found on stray word in some docs but otherwise things seem perfect.

[hc-github-team-consul-core]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/361903.

[hc-github-team-consul-core]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/361915.

[hc-github-team-consul-core]

🍒✅ Cherry pick of commit 3ad754c onto release/1.10.x succeeded!