kvserver: gossip store read amplification

[kvoli]

Previously, other store statistics were gossipped between stores to
inform allocation and rebalancing decisions. This patch introduces an
additional gosipped value, the store read amplification. This patch
allows the allocator to avoid rebalancing ranges onto stores with poor
lsm health, indicated by a large read amplification.

resolves #74254

Release note: None

[cockroach-teamcity]

This change is 

[aayushshah15]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @kvoli)

---

pkg/roachpb/metadata.proto, line 334 at r1 (raw file):

  optional double writes_per_second = 5 [(gogoproto.nullable) = false];
  // read_amplification tracks the current read amplification in the store.
  optional int64 read_amplification = 11 [(gogoproto.nullable) = false];

We should add a TODO here or somewhere else that mentions that the use of this field will need to be version-gated.

[kvoli]

Thanks for the review.

  optional double writes_per_second = 5 [(gogoproto.nullable) = false];
  // read_amplification tracks the current read amplification in the store.
  optional int64 read_amplification = 11 [(gogoproto.nullable) = false];

We should add a TODO here or somewhere else that mentions that the use of this field will need to be version-gated.

That sounds like a good idea. What would be the process here with mixed versions clusters? Does version gating ensure no overlapping of mismatched versions occurs?

[aayushshah15]

What would be the process here with mixed versions clusters?

When we extend the allocator logic to use this new attribute, it will need to ensure that the current cluster version is one that supports this new ReadAmplication attribute (in other words, it will need to ensure that there are no older nodes in the cluster that aren't broadcasting this new attribute). For a random example, see here:

cockroach/pkg/kv/kvserver/replicate_queue.go

Lines 554 to 555 in 208e2b4

	if !repl.store.cfg.Settings.Version.IsActive(ctx,
	clusterversion.EnableLeaseHolderRemoval) {

If this check fails, that means we're in a mixed version state and we can't use this new attribute.

[kvoli]

bors r+

[craig]

Build failed:

• GitHub CI (Cockroach)

[kvoli]

bors r-

[kvoli]

Build failed:

* [GitHub CI (Cockroach)](https://teamcity.cockroachdb.com/viewLog.html?buildId=4467367&buildTypeId=Cockroach_UnitTests)

Rebased on master, going to re-run the CI here.

[kvoli]

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)