roachperf: regression around 2022-06-27

[erikgrinaker]

There's a roachperf regression around 2022-06-27, e.g. on kv95/enc=false/nodes=3/cpu=32:

This is primarily due to #82045, which merged on 2022-06-26, but I'm also seeing slowdowns later:

Commit	Date	KV95
b47dc37	2022-06-25	123605.7
460ce6a	2022-06-26	119619.3
e1e99da	2022-06-30	114801.3
d324df1	2022-07-01	114091.8
9c5472e	2022-07-05	113846.0

Interestingly, I'm not seeing the giant drop at the end. Might just be a fluke, or temporary change, although I do see it across a few benchmarks.

Jira issue: CRDB-17337

Epic CRDB-2624

[erikgrinaker]

Bisected the second regression down to #78085 on 2022-06-28, so that's also related to MVCC range tombstones. Dumping some raw data from bisection:

9c5472e9dc 2022-07-05: 108152.7 5693.3 113846.0
d324df18a1 2022-07-01: 108392.4 5699.4 114091.8
e1e99da678 2022-06-30: 109058.6 5742.7 114801.3
a57d870be9 2022-06-28: 109213.9 5746.6 114960.5
e19d98d6a6 2022-06-28: 109560.5 5765.8 115326.2
8fe2362696           : 109763.4 5770.2 115533.7 (MVCCStats)
5dc6228030 2022-06-28: 113632.8 5977.1 119610.0
0c3f215764           : 114198.3 6011.5 120209.8
460ce6ad67 2022-06-26: 113635.0 5984.2 119619.3 (scan/get enabled)
b47dc37441 2022-06-25: 117423.1 6182.6 123605.7

[blathers-crl]

cc @cockroachdb/replication

[nicktrav]

We're back to around 116k. What's the target for calling this one done? @erikgrinaker @jbowens

[erikgrinaker]

Going to run some targeted benchmarks and get some numbers first, but we can likely close this out this week.

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

I ran some kv95/enc=false/nodes=1/cpu=32 runs comparing master to v22.1.5, since that's the only benchmark that's still significantly below the previous levels. However, these both showed results in the same ballpark. The CPU profiles also had MVCCGet pretty far down on the list, at about 7% cumulative. So it isn't clear that this is even caused by the MVCC work.

I'm going to close this one out for now, since we're recovered most of the regression in multi-node benchmarks. We should still run some comparisons between 22.1 and master, but this isn't the right issue to do so.