core: profile IOPs usage

[awoods187]

Currently, the managed service on AWS uses EBS volumes. These volumes are priced based on IOPs usage and as such, IOPs are one of the biggest costs to providing this service.

We should profile our IOPs usage to make sure that it meets our expectations and look for opportunities to reduce its usage.

[awoods187]

cc @petermattis and @kannanlakshmi

[petermattis]

Per in-person discussion: RocksDB background operations are a significant source of IOPs. Improvements to compaction heuristics there, or in Pebble, have the potential to reduce IOPs, though such improvements are still on the drawing board.

#38322 might significantly reduce IOPs. Cc @nvanbenschoten

[kannanlakshmi]

Posting breakdown of MSO costs (second tab) - as you can see storage and IOPS specifically are the highest driver of cost which will scale (I assume) as the cloud business grows. Any work in that direction will be directly beneficial in the long term profitability of the cloud business.

https://docs.google.com/spreadsheets/d/1OxSNnQQMvy3j7V8IbsHB-Et7NHwM5yLpo8tpXag7vyE/edit#gid=0

[andreimatei]

I think issues like this (let's see if we're as good as we can be and if we're not let's become better) don't serve much purpose. At the very least it should start with someone's surprise at some data.
We have too many issues already... Mind if I close it? :)

[awoods187]

I disagree--this issue is to make sure we do the investigation. Without an issue we don't have a central place to record that and put the motivation for it.

[ajwerner]

After talking with @bdarnell I did some investigation of the value of IOPs using io1 in AWS a couple of weeks ago with a specific focus on YCSB/A with a uniform distribution (roughly KV50 with larger values). The results indicate that we may be over-provisioning IOPs and likely warrant further investigation.

See the data here:
https://docs.google.com/spreadsheets/d/1VjVCZmnHGGpxtjbp3lIllZOpsCc7S29cSj_gZacWFF0/edit?usp=sharing (internal only)

[petermattis]

@ajwerner Interesting data. How long did you run the tests for? For short runs, background compactions will be insignificant which might allow lower IOPs to perform well.

[ajwerner]

The test was run for 10 minutes each. Also worth noting that the data in the sheet is the average from 5 runs each. #38161 has the logic for the roachtest.

[petermattis]

@ajwerner I'd guess that isn't long enough to hit significant RocksDB background operations. Data is still interesting, but let's be careful about reading too much into it.

[awoods187]

I think we should consider adding information like this to TPC-C testing as a fail condition (once we establish a baseline).

To reiterate, I don't think this issue is about identifying the "correct" amount of IOPs usage for all workloads--it is about testing some common workloads and understanding where IOPs is consumed as well as determining if the consumption matches our expectation as a measurement of the DB.

cc @andy-kimball

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!