kvserver: add pebble ingestion/flush metrics

[tbg]

This adds:

storage.l0-bytes-flushed
storage.lX-num-files
storage.lX-bytes-ingested

which are all useful to understand inverted LSMs.

Release note: None

[cockroach-teamcity]

This change is 

[sumeerbhola]

Reviewed 2 of 2 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @irfansharif and @tbg)

---

pkg/kv/kvserver/metrics.go line 465 at r1 (raw file):

	"Number of SSTables in Level %d",
	"SSTables",
	metric.Unit_COUNT,

How will we use the number of files in the other levels?
The L0 files are exported because they are used in admission control, and now also as a compaction signal for L0.
That is not the case for the other levels.

---

pkg/kv/kvserver/metrics.go line 1871 at r1 (raw file):

		AverageQueriesPerSecond: metric.NewGaugeFloat64(metaAverageQueriesPerSecond),
		AverageWritesPerSecond:  metric.NewGaugeFloat64(metaAverageWritesPerSecond),
		// TODO(tbg): this histogram seems bogus? What are we tracking here?

I think it is part of the allocator decision making
@kvoli would know.

---

pkg/kv/kvserver/metrics.go line 1906 at r1 (raw file):

		RdbPendingCompaction:        metric.NewGauge(metaRdbPendingCompaction),
		RdbMarkedForCompactionFiles: metric.NewGauge(metaRdbMarkedForCompactionFiles),
		RdbL0BytesFlushed:           metric.NewGauge(metaRdbL0BytesFlushed),

Flushed and ingested are cumulative values, i.e., not gauges. I guess we are having to do this because of the deficiency of our metrics system.
We should at least have a code comment. A bunch of the existing metrics, *Hits, *Misses, *Count are also cumulative.

[tbg]

Dismissed @sumeerbhola from 3 discussions.
Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @irfansharif and @sumeerbhola)

---

pkg/kv/kvserver/metrics.go line 465 at r1 (raw file):

Previously, sumeerbhola wrote…

How will we use the number of files in the other levels?
The L0 files are exported because they are used in admission control, and now also as a compaction signal for L0.
That is not the case for the other levels.

I removed all but L0.

---

pkg/kv/kvserver/metrics.go line 1906 at r1 (raw file):

Previously, sumeerbhola wrote…

Flushed and ingested are cumulative values, i.e., not gauges. I guess we are having to do this because of the deficiency of our metrics system.
We should at least have a code comment. A bunch of the existing metrics, *Hits, *Misses, *Count are also cumulative.

I added a comment.

[tbg]

bors r=sumeerbhola

TFTR!

[craig]

Build failed:

• GitHub CI (Cockroach)

[tbg]

bors r=sumeerbhola

TestLeaseholdersRejectClockUpdateWithJump
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLeaseholdersRejectClockUpdateWithJump1654522692
    test_log_scope.go:80: use -show-logs to present logs inline
    client_replica_test.go:182: remote wall time is too far ahead (500.007519ms) to be trustworthy

#79641

[craig]

Build succeeded:

• GitHub CI (Cockroach)

[irfansharif]

blathers backport 22.1

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 8dfd075 to blathers/backport-release-22.1-81039: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.1 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[irfansharif]

@tbg, these metrics would've come in handy in this internal escalation. They were running 22.1, these metrics are only present in 22.2+. Could we backport?