kvserver: re-evaluate raft log truncation heuristics

[tbg]

Is your feature request related to a problem? Please describe.

The raft log truncation heuristics date back to a world in which we were very aggressive about truncations (since the log needed to be included in snapshots and could make them very large) while simultaneously being very defensive (if a snapshot is in flight, we will never cut it off from the log).

We just saw (internal link) that these heuristics could be problematic, and that some of the thresholds were not adjusted when we changed our default range size from 64mb to 512mb.

We should revisit all of these heuristics and, through thought and experimentation, adjust them with the goal of avoiding pathological build-up of raft log as well as "unnecessary" rejection of snapshots

Additional context

#36262 also proposes allowing followers to truncate "locally" (i.e. without being prompted to truncated by the leader), in which case they would use different (simpler) heuristics than the leader.

Jira issue: CRDB-12843

Epic CRDB-39898

[erikgrinaker]

Additionally, #75793 adds a raftLogTruncationClearRangeThreshold constant that controls the number of entries beyond which log truncation switches from dropping Pebble point tombstones to range tombstones. This is set very conservatively (100000), and is unlikely to trigger even with lagging followers and hitting RaftLogTruncationThreshold (16 MB), due to concerns that dropping too many range tombstones may be detrimental to Pebble.

A setting of e.g. 1000 appears more reasonable here, but this needs to be tuned in concert with e.g. RaftLogTruncationThreshold and other parameters, and benchmarked to make sure the range tombstones are an improvement rather than a detriment in most or all realistic circumstances. Also related to cockroachdb/pebble#1295.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!