Re-enable stats-based replica rebalancing

[jseldess]

This means turning on both load-based replica rebalancing and size-based replica rebalancing. So in addition to some how-to docs, we'll need to update our architecture docs.

Background: https://airtable.com/tblD3oZPLJgGhCmch/viwINAgXA5on9pr2T/recWJ095g7CriLJ88
PM: @awoods187
Eng Partner: @a-robinson

[jseldess]

PR: cockroachdb/cockroach#28340

From release notes:

Range leases are now automatically rebalanced throughout the cluster to even out the amount of QPS being handled by each node. [#28340][#28340] {% comment %}doc{% endcomment %}

@a-robinson, is this work now done and ready to be documented?

[a-robinson]

Not quite, I'd suggest waiting until next week.

[jseldess]

From forum:

• The decisions are based on the number of queries processed by a node made over the last half hour or so using an exponentially weighted moving average to more heavily weight recent load. They only happen if load on the cluster is above a certain threshold, the difference in load between nodes is substantial (greater than 25% difference by default), and there are leases that could be transferred that will meaningfully even out the load in the cluster.
• Only leases are transferred based on load as of today. Very little I/O is involved in lease transfers, since the data itself doesn’t have to be moved from the nodes it’s currently on in order to change the leaseholder.