kvserver: use simulator to evaluate 23.2 allocation changes

[shralex]

We should use the Simulator to evaluate any 23.2 changes, related to MR or other.

Jira issue: CRDB-32088

[blathers-crl]

Hi @shralex, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[kvoli]

I went through allocator related changes since 2023-03-01 on the release-23.2 branch and flagged any that would be candidates for additional simulator testing:

• ed51752
• 8478b85
• 1fac2f8
• 9ed7027
• 3e3a9b1
• a944e57

[kvoli]

Mapping the commits to their respective pull requests and adding some additional information on the change, and level of testing (inc simulator) currently:

• allocator: prioritize non-voter promotion to satisfy voter constraints #111609
  • asim: enhance asim testing framework with randomized testing integration  #106192 extended the simulator testing for added coverage of constraint violation. The issue isn't complete, but the changes have been evaluated reasonably well with the existing simulator changes done here.
• kvserver: pass Desc and SpanConf through allocator #108197
  • PR changes desc/span config to be passed through the allocator, instead of called ad-hoc when needed. Added testing coverage could be modifying the descriptor underneath repeatedly and checking for spurious results. This is a low priority candidate.
• kvserver: transfer lease when acquiring lease outside preferences #107507
  • There is no lease preference testing in the simulator currently. This is a good candidate. The conformance assertions include constraints, under(over) replication and unavailable ranges. It would be reasonable to extend the conformance assertions to include lease preferences.
• allocator: promote non-voter in 1-voter rebalance #106202
  • Coverage was added for the 1-replica case (to some extent) via asim: enhance asim testing framework with randomized testing integration  #106192.
• allocator: expose LeaseRebalanceThreshold as a cluster setting #105924
  • Made a threshold a cluster setting. This could be verified further via an explicit datadriven test which modifies the cluster setting, this is lower relative priority.
• kvserver: increase rebalance action impact req #98889
  • Good candidate for testing. A bad distribution after this change is a bimodal distribution with many ranges. Where there are equally hot ranges (n>1/threshold), and dormant ranges. In this case, the store rebalancer could never kick in to move load, where it previously would. The benefit of this change is greater stability.