kvserver,allocator: overloaded store with no leases is unable to shed load

[kvoli]

In #106140 we saw a hot node which couldn't shed any load at one point due to not having any leases i.e. follower overload.

The store rebalancer is responsible for helping relieve overload, however it only operates on ranges it has a lease for.

cockroach/pkg/kv/kvserver/store_rebalancer.go

Lines 728 to 731 in 1d3c11e

	if !candidateReplica.OwnsValidLease(ctx, now) {
	log.KvDistribution.VEventf(ctx, 3, "store doesn't own the lease for r%d", candidateReplica.GetRangeID())
	continue
	}

cockroach/pkg/kv/kvserver/store_rebalancer.go

Lines 930 to 933 in 1d3c11e

	if !candidateReplica.OwnsValidLease(ctx, now) {
	log.KvDistribution.VEventf(ctx, 3, "store doesn't own the lease for r%d", candidateReplica.GetRangeID())
	continue
	}

The replicate queue will balance the #replicas, however this store already had fewer replicas than others.

The result is, once in this state, the node will never shed sufficient load to stop being overloaded.

Jira issue: CRDB-30169