cloud: add read and write metrics

[adityamaru]

This change adds cloud.read_bytes and cloud.write_bytes metrics.
These are updated on all read/write operations to external storage endpoints.

These metrics while shared by all external storage interactions
will provide an immediate signal into whether we're seeing an
abnormal rate of reads and writes errors during support investigations.

Fixes: #89242

Release note: None

[cockroach-teamcity]

This change is 

[adityamaru]

A 2TB import; I paused it in the middle to fiddle with some cluster settings for funsies.

And then a backup of that data:

[dt]

Are we concerned that the retry count is potentially not going to reflect retries in the implementation of some clients and/or at a higher layer either? A misleading metric might be worse than no metric eg if someone looks at it, sees it is flat and dismisses retries even if they're happening inside foo-sdk or library?

[adityamaru]

Are we concerned that the retry count is potentially not going to reflect retries in the implementation of some clients and/or at a higher layer either?

Yeah that's a good point and one that crossed my mind when I was plumbing counter increments into all the retry loops. We could try to be diligent about having all clients read and write paths that we retry in pkg/cloud/<provider> increment this counter but maybe that's too optimistic. This doesn't address internal SDK retries or higher-level job retries either but maybe tweaking the metric description to make this clear solves that?

Alternatively, we could breakdown the retries into cloud.resumingreader.error.retries, cloud.custom.error.retries to better define what they track? The latter being any custom error handling we inject into the SDKs.

[adityamaru]

TFTR!

bors r=dt

[craig]

Merge conflict.

[adityamaru]

bors r=dt,stevendanna

[adityamaru]

bors retry

[adityamaru]

bors r-

[craig]

Canceled.

[adityamaru]

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 17a5159 to blathers/backport-release-22.2-89584: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}