Radix-sorted, lookup-free, typesafe, nonblocking, priority-boosting pending tasks

[DaveCTurner]

Today the master's pending task queue is just the
PriorityBlockingQueue<Runnable> belonging to the underlying
ThreadPoolExecutor. The reasons for this date back a long way but it
doesn't really reflect the structure of the queue as it exists today. In
particular, we must keep track of batches independently of the queue
itself, and must do various bits of unchecked casting to process
multiple items of the same type at once.

This commit introduces an new queueing mechanism, independent of the
executor's queue, which better represents the conceptual structure of
the master's pending tasks:

• Today we use a priority queue to allow important tasks to preempt
  less-important ones. However there are only a small number of priority
  levels, so it is simpler to maintain a queue for each priority,
  effectively replacing the sorting within the priority queue with a
  radix sort.

• Today when a task is submitted we perform a map lookup to see if it
  can be added to an existing batch or not. With this change we allow
  client code to create its own dedicated queue of tasks. The entries in
  the per-priority-level queues are themselves queues, one for each
  executor, representing the batches to be run.

• Today each task in the queue holds a reference to its executor, but
  the executor used to run a task may belong to a different task in the
  same batch. In practice we know they're the same executor (that's how
  batches are defined) but we cannot express this knowledge in the type
  system so we have to do a bunch of unchecked casting to work around it.
  With this change we associate each per-executor queue directly with its
  executor, avoiding the need to do all this unchecked casting.

• Today the master service must block its thread while waiting for each
  task to complete, because otherwise the executor would start to process
  the next task in the queue. This makes testing using a
  DeterministicTaskQueue harder (see FakeThreadPoolMasterService).
  With this change we avoid submitting a task to the ThreadPoolExecutor
  until the previous task is complete, which means we can make the
  implementation avoid blocking while a task is running and therefore run
  the whole production implementation in deterministic tests¹.

• Today it's possible for a steady drip of high-priority tasks to starve
  low-priority tasks of access to the master for an extended period of
  time. With this change we separate the queue of tasks from the queue
  which determines the execution order. This allows us to implement more
  intelligent execution policies. For instance, if we detect that the
  queue has not processed any low-priority tasks for too long then we can
  make the decision to boost their priorities².

Relates #81626

Footnotes

1. Not done yet but this is a step in the right direction. ↩

2. Not done yet but this is a step in the right direction. ↩

[DaveCTurner]

Superseded by #92021