Test that qps doesn't dip when gracefully draining a node

[a-robinson]

This is an important scenario that could really use some regression test coverage, as indicated by the fact that nobody noticed or followed up on #22573 until more than 4 months after 1.1 was released.

This seems like a good fit to be a workload test -- run something like kv with its -max-rate flag set, then gracefully stop a node and expect QPS to not dip more than a few percent below the specified -max-rate. If we wanted to make this extra rigorous, #23202 could be used to pin all leases on the node that we stop before we stop it.

cc @asubiotto

[tbg]

+1, this seems pretty important. One requirement here is being able to programmatically obtain the load generator statistics (ideally while the load runs). I wonder if workload should export an HTTP interface for that. Or we can query the cluster statement statistics (this is nice because users should be able to access this information for their workload, too). Or workload could insert periodically into a statistics table that we can then query.

[petermattis]

Do you need cluster statement statistics, or access to some of the internal time series metrics? For a specific metric, the time series are already programmatically available (though the specific magic incantation is a bit involved).

[tbg]

I think we'd want to be able to run a few different load gens eventually, and some of them might be so low in qps that their dip could be shadowed by a faster one (say kv). Maybe statement statistics can do well enough for starters.

[petermattis]

I think we'd want to be able to run a few different load gens eventually, and some of them might be so low in qps that their dip could be shadowed by a faster one (say kv).

Good point, though for initial test a single QPS metric would suffice.

Maybe statement statistics can do well enough for starters.

Yeah. I've forgotten the specifics of when these are reset and the info they contain, but certainly seems possible they could work.

[nvanbenschoten]

Most of this was addressed by #26542, which gracefully shuts down a third of a cluster and watches the QPS of kv. It asserts that QPS did not drop by more than 20%. Is there anything more to this issue that we should address for 2.1?

[petermattis]

Is there anything more to this issue that we should address for 2.1?

I think that test is sufficient.

[a-robinson]

I don't think that test actually tests this? That test is basically:

1. Run some load and measure the qps
2. Stop the load
3. Stop a node
4. Run some load and measure the qps
5. Stop the load
6. Compare the qps results

It is explicitly not trying to measure how an ongoing load is affected by the process of draining a node, and almost certainly would not have found the various bugs in the node draining logic that motivated this issue.