storage: raftLogQueue locking causes contention on large machines

[ajwerner]

A mutex profile on a 3-node 32-core high CPU cluster running KV0 with 1024 splits (I ran it with a variety of concurrencies) reveals that the vast majority of time is spent waiting on the raftLogQueue.maybeAdd.

It seems possible to reconsider this locking in a targeted way given that a CPU profile for this looks like it's mostly accessing system and replica state rather than queue state.

[tbg]

#36262 should be of service with that because it can help us remove that queue altogether.

[tbg]

BTW, I'm curious how this looks on 19.2+, since I made that change to make adding to the queues async. (The work is still being done which is annoying, but at least it won't be blocking the hot path any more).

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!