roachtests: introduce admission-control/snapshot-overload

[irfansharif]

This test sets up a 3-node CRDB cluster on 8vCPU machines, loads it up
with a large TPC-C dataset, and sets up a foreground load of kv95/1b.
The TPC-C dataset has a replication of factor of 1 and is used as the
large lump we bus around through high-rate snapshots -- snapshots that
are send to the node containing leaseholders serving kv95 traffic.
Snapshots follow where the leases travel, cycling through each node,
evaluating performance isolation in the presence of snapshots.
Informs #80607.

This test comes 'battery-included', shipping with a custom grafana
dashboard and requisite prometheus/grafana plumbing. At the end of test
run a prometheus dump is kept around for later inspection. It also
integrates with --local and --skip-init for fast iteration. We're going
to cargo cult such things in future integration tests for admission
control.

Release note: None

---

There are a few other miscellaneous cleanups/conveniences around the roachtest package for @cockroachdb/test-eng to look at. First three commits are from #89027 and don't need to be looked at here.

[cockroach-teamcity]

This change is 

[irfansharif]

02:44:38 admission_control_snapshot_overload.go:114: test status: initializing kv dataset (<1m0s)
02:44:42 admission_control_snapshot_overload.go:127: test status: initializing tpcc dataset (<20m0s)
02:57:04 admission_control_snapshot_overload.go:151: test status: starting kv workload thread to run for 1h40m0s (<1m0s)
02:57:04 admission_control_snapshot_overload.go:166: test status: setting performance baseline (<5m0s)
03:02:04 admission_control_snapshot_overload.go:169: test status: starting snapshot transfers for 1h30m0s (<1m0s)
03:02:04 admission_control_snapshot_overload.go:194: test status: waiting for workload/snapshot transfers to finish (<1h35m0s)
03:02:04 admission_control_snapshot_overload.go:173: test status: snapshot round 1/3: inert data and active leases routing to n2 (<30m0s)
03:32:04 admission_control_snapshot_overload.go:173: test status: snapshot round 2/3: inert data and active leases routing to n3 (<30m0s)
04:02:04 admission_control_snapshot_overload.go:173: test status: snapshot round 3/3: inert data and active leases routing to n1 (<30m0s)

Here's a prometheus dump of an experiment run. To see it, run the contained ./prometheus-docker-run.sh and point to localhost:9090 through your grafana instance (brew install grafana; brew services start grafana + localhost:3000 if you're on macs). Screenshots:

snapshot-overload.zip

[irfansharif]

I’ll echo the same learnings we had a few months ago — it’s difficult to get into resource starvation due to snapshots, I’ve had to crank things way up to get anything measurable. And even what's measurable is tiny -- a p99 elevation from 23ms to 55ms with foreground load running against a fully loaded LSM observing a lot of incoming snapshots. It's possible this test gets tweaked slightly (using tpcc as the foreground load, or higher read rate, etc.) once we're ready to look at the snapshot overload case again. Perhaps once we're looking to remove snapshot rates altogether.

[sumeerbhola]

Are those high cpu and read/write bandwidth plateaus due to snapshot ingestion?

[irfansharif]

Are those high cpu and read/write bandwidth plateaus due to snapshot ingestion?

Going to take the chance to re-visualize off of the prometheus dump to plot all relevant things onto one frame, focusing on one node in particular. For read-write bandwidth, the two non-zero plateaus correspond to when we're:

• actively ingesting snapshots + compacting
• just compacting post snapshot ingestion

For CPU, the six non-zero plateaus correspond to when:

• a node without the TPC-C dataset acquires leases for KV data
• it starts ingesting + compacting
• it starts compacting
• it sheds leaseholders to the next node
• it's done compacting
• it starts sending snapshots to the next node

[sumeerbhola]

Thanks for the details.

for the admission_control* files

Reviewed 3 of 17 files at r9.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @irfansharif, @renatolabs, and @sumeerbhola)

[irfansharif]

bors r+

@renatolabs, feel free to leave more comments if you have any, I'll send a follow up PR. Trying to get this in to unblock some other follow on PRs.

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)