Merge #110057 #110818 #111039

110057: kvserver: unskip and deflake replicate queue dead non voters r=andrewbaptist a=kvoli

TestReplicateQueueDeadNonVoters would flake periodically due to racing
between the snapshot and replicate queue.

Add a filter to queue processing in order to only process the scratch
range. This speeds up the test and prevents timing conditions between
the two queues under stress. The test remains skipped under stress-race
due to its size (5 nodes) causing timeouts.

Resolves: #76996
Epic: none
Release note: None

110818: kvserver: export secondary cache counter metrics r=joshimhoff a=joshimhoff

Fixes #110899.

**kvserver: export secondary cache counter metrics**

This commits exports counter metrics regarding the secondary cache added at cockroachdb/pebble#2760. This commit doesn't export the histogram metrics added in that PR. While working on this one, I have realized a follow up PR is needed to export the bucketing scheme to CRDB.

Release note: None.

111039: sql: fix flaky TestHotRangesStats r=j82w a=j82w

The test failed with timeouts under stress. The test is switched to use the SucceedsSoon which allows for more time when under stress.

Fixes: #111036

Release note: None

Co-authored-by: Austen McClernon <[email protected]>
Co-authored-by: Josh Imhoff <[email protected]>
Co-authored-by: j82w <[email protected]>

docs/generated/metrics/metrics.html

@@ -654,6 +654,16 @@
 <tr><td>STORAGE</td><td>storage.l6-level-size</td><td>Size of the SSTables in level 6</td><td>Bytes</td><td>GAUGE</td><td>BYTES</td><td>AVG</td><td>NONE</td></tr>
 <tr><td>STORAGE</td><td>storage.marked-for-compaction-files</td><td>Count of SSTables marked for compaction</td><td>SSTables</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
 <tr><td>STORAGE</td><td>storage.queue.store-failures</td><td>Number of replicas which failed processing in replica queues due to retryable store errors</td><td>Replicas</td><td>COUNTER</td><td>COUNT</td><td>AVG</td><td>NON_NEGATIVE_DERIVATIVE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.count</td><td>The count of cache blocks in the secondary cache (not sstable blocks)</td><td>Cache items</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.evictions</td><td>The number of times a cache block was evicted from the secondary cache</td><td>Num evictions</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-full-hit</td><td>The number of reads where all data returned was read from the secondary cache</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-multi-block</td><td>The number of secondary cache reads that require reading data from 2+ cache blocks</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-multi-shard</td><td>The number of secondary cache reads that require reading data from 2+ shards</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-no-hit</td><td>The number of reads where no data returned was read from the secondary cache</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-partial-hit</td><td>The number of reads where some data returned was read from the secondary cache</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.reads-total</td><td>The number of reads from the secondary cache</td><td>Num reads</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.size</td><td>The number of sstable bytes stored in the secondary cache</td><td>Bytes</td><td>GAUGE</td><td>BYTES</td><td>AVG</td><td>NONE</td></tr>
+<tr><td>STORAGE</td><td>storage.secondary-cache.write-back-failures</td><td>The number of times writing a cache block to the secondary cache failed</td><td>Num failures</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>
 <tr><td>STORAGE</td><td>storage.shared-storage.read</td><td>Bytes read from shared storage</td><td>Bytes</td><td>GAUGE</td><td>BYTES</td><td>AVG</td><td>NONE</td></tr>
 <tr><td>STORAGE</td><td>storage.shared-storage.write</td><td>Bytes written to external storage</td><td>Bytes</td><td>GAUGE</td><td>BYTES</td><td>AVG</td><td>NONE</td></tr>
 <tr><td>STORAGE</td><td>storage.wal.bytes_in</td><td>The number of logical bytes the storage engine has written to the WAL</td><td>Events</td><td>GAUGE</td><td>COUNT</td><td>AVG</td><td>NONE</td></tr>