i#6822 unscheduled: Add unscheduled-input drmemtrace support (#6826)

Augments the drmemtrace scheduler with a new notion of an "unscheduled"
input which, if it has no timeout, is not runnable indefinitely until
another input explicitly wakes it up.

Adds 3 new markers:
+ TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE, which makes the caller
"unscheduled".
+ TRACE_MARKER_TYPE_SYSCALL_SCHEDULE, which makes a target no longer
"unscheduled".
+ TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT which adds a timeout parameter
to TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE and
TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH

Adds handling for the new marker types.

Changes TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH to make the source input
unscheduled, unless a timeout is passed which is honored (previously it
was ignored and the syscall latency was used as the block time if it was
over the thresholds).

Adds a fallback in case a state is reached with no schedulable inputs
yet some number of unscheduled inputs who would otherwise hang forever.

Adds unit tests.

Also tested on a large trace with many real-world cases of these direct
switches and unschedulable threads.

Issue: #6822

clients/drcachesim/common/trace_entry.h

@@ -567,10 +567,21 @@ typedef enum {
      * #TRACE_MARKER_TYPE_SYSCALL and #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL markers)
      * that causes an immediate switch to another thread on the same core (with the
      * current thread entering an unscheduled state), bypassing the kernel scheduler's
-     * normal dynamic switch code based on run queues.  The marker value holds the
-     * thread id of the target thread.  This should generally always be after a
-     * #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL marker as such a switch always
-     * has a chance of blocking if the target needs to be migrated.
+     * normal dynamic switch code based on run queues.  The marker value holds the thread
+     * id of the target thread.  The current thread will remain unschedulable
+     * indefinitely unless another thread resumes it with either
+     * #TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH or #TRACE_MARKER_TYPE_SYSCALL_SCHEDULE;
+     * or, if a #TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT marker is present, the thread will
+     * become schedulable when that timeout expires.  This marker provides a mechanism to
+     * model these semantics while abstracting away whether the underlying system call is
+     * a custom kernel extension or a variant of "futex" or other selective wait-notify
+     * scheme.  This marker should generally always be after a
+     * #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL marker as such a switch always has a
+     * chance of blocking the source thread.  See also
+     * #TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT, #TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE, and
+     * #TRACE_MARKER_TYPE_SYSCALL_SCHEDULE.  The scheduler only models this behavior when
+     * #dynamorio::drmemtrace::scheduler_tmpl_t::scheduler_options_t.honor_direct_switches
+     * is true.
      */
     TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH,
 
@@ -627,6 +638,51 @@ typedef enum {
      */
     TRACE_MARKER_TYPE_VECTOR_LENGTH,
 
+    /**
+     * This marker is emitted prior to a system call (but after the system call's
+     * #TRACE_MARKER_TYPE_SYSCALL and #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL markers)
+     * that causes the current thread to become unschedulable (removed from all queues of
+     * runnable threads).  The thread will remain unschedulable indefinitely unless
+     * another thread resumes it with either #TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH or
+     * #TRACE_MARKER_TYPE_SYSCALL_SCHEDULE; or, if a
+     * #TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT marker is present, the thread will become
+     * schedulable when that timeout expires.  This marker provides a mechanism to model
+     * these semantics while abstracting away whether the underlying system call is a
+     * custom kernel extension or a variant of "futex" or other selective wait-notify
+     * scheme.  This marker should generally always be after a
+     * #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL marker as becoming unschedulable is a
+     * form of blocking and results in a context switch.  The scheduler only models this
+     * behavior when
+     * #dynamorio::drmemtrace::scheduler_tmpl_t::scheduler_options_t.honor_direct_switches
+     * is true.
+     */
+    TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE,
+
+    /**
+     * This marker is emitted prior to a system call (but after the system call's
+     * #TRACE_MARKER_TYPE_SYSCALL marker) that causes a target thread identified in the
+     * marker value to become schedulable again if it were currently unschedulable or if
+     * it is not currently unschedulable to *not* become unschedulable on its next action
+     * that would otherwise do so.  See also #TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE and
+     * #TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH.  This marker provides a mechanism to
+     * model these semantics while abstracting away whether the underlying system call is
+     * a custom kernel extension or a variant of "futex" or other selective wait-notify
+     * scheme.  The scheduler only models this behavior when
+     * #dynamorio::drmemtrace::scheduler_tmpl_t::scheduler_options_t.honor_direct_switches
+     * is true.
+     */
+    TRACE_MARKER_TYPE_SYSCALL_SCHEDULE,
+
+    /**
+     * This marker is emitted prior to a system call (but after the system call's
+     * #TRACE_MARKER_TYPE_SYSCALL and #TRACE_MARKER_TYPE_MAYBE_BLOCKING_SYSCALL markers)
+     * which also has a #TRACE_MARKER_TYPE_DIRECT_THREAD_SWITCH or
+     * #TRACE_MARKER_TYPE_SYSCALL_UNSCHEDULE marker.  This indicates a timeout provided
+     * by the application after which the thread will become schedulable again.  The
+     * marker value holds the timeout duration in microseconds.
+     */
+    TRACE_MARKER_TYPE_SYSCALL_ARG_TIMEOUT,
+
     // ...
     // These values are reserved for future built-in marker types.
     // ...