Event loop: update the rendering step mismatches implementations in subtle ways

[noamr]

What is the issue with the HTML Standard?

There are a few places in the event loop processing model, specifically in its relationship with updating the rendering, that differs from how it's implemented in the difference browser. This is web observable in subtle ways:

• The timestamp for rAF comes from vsync in Gecko/chromium, but is the time at the beginning of the update the rendering phase in webkit
• There is no "task" during the update the rendering phase in the spec (see How can microtasks run without a task? #7845). In Chromium, there is a special task for rendering (In Gecko too I think, but not familiar enough). In WebKit, the concept of tasks is not 100% integrated. This is observable e.g. in APIs like longtasks.
• It's unclear when UI events are handled, except for scroll/resize. In implementations those are somewhat coupled with the rendering "task".

[annevk]

I think we should make "update the rendering" its own task with its own task source. The implicit task source prioritization algorithm would then attempt to make that task hit the desired frame rate.

The issue with UI events has been known for a long time. w3c/uievents#238 (comment) describes (rather briefly) what needs to be done: we need to turn the OS inputs into abstract algorithms from which we integrate with the event loop, create the relevant event objects, initialize their fields correctly, handle dispatch and cancelation, etc.

cc @rniwa @smaug----

[Kaiido]

• For the first point, this is already confusing developers since this can cause the reported timestamp to be at a point earlier than when the call to requestAnimationFrame has been done, see e.g. https://stackoverflow.com/questions/64177381/timestamp-of-requestanimationframe-is-not-reliable/ or https://stackoverflow.com/q/38360250/3702797.
  According to this comment having it fixed to the VSync signal timestamp would be the most useful for users, but then it gets a bit more problematic to spec for when there is no VSync signal, adaptive frame rate, or even no monitor at all.

• For the second point, I remember I read that Firefox also uses a task for update the rendering, but I didn't know it's observable. Could you share an example test? That's indeed gonna be more and more important with stuff like prioritized task scheduling.

• For the last point, indeed UI events are ill-defined. It seems that you are referring to the fact mouse events are throttled to the monitor's frame rate though, which is in accordance to what UI events ask for, e.g. for mousemove:

The frequency rate of events while the pointing device is moved is implementation-, device-, and platform-specific, but multiple consecutive mousemove events SHOULD be fired for sustained pointer-device movement, rather than a single event for each instance of mouse movement. Implementations are encouraged to determine the optimal frequency rate to balance responsiveness with performance.

But then again, there is the pointerawupdate event that should not be throttled the same way. Probably having clearer hooks, or even more inclusion of UI events into HTML would help here.

[noamr]

• For the first point, this is already confusing developers since this can cause the reported timestamp to be at a point earlier than when the call to requestAnimationFrame has been done, see e.g. https://stackoverflow.com/questions/64177381/timestamp-of-requestanimationframe-is-not-reliable/ or https://stackoverflow.com/q/38360250/3702797.
  According to this comment having it fixed to the VSync signal timestamp would be the most useful for users, but then it gets a bit more problematic to spec for when there is no VSync signal, adaptive frame rate, or even no monitor at all.

The way I would spec it in a way that's closer to implementations is that there are "in parallel" step that continuously wait for a rendering opportunity, save the timestamp when that rendering opportunity was available, and then the main event loop would read that timestamp (or some such). VSync or not etc. would be more implementation defined.

• For the second point, I remember I read that Firefox also uses a task for update the rendering, but I didn't know it's observable. Could you share an example test? That's indeed gonna be more and more important with stuff like prioritized task scheduling.

Yes, this is observable in longtasks:
https://wpt.fyi/results/longtask-timing/longtask-in-raf.html?label=experimental&label=master&aligned

• For the last point, indeed UI events are ill-defined. It seems that you are referring to the fact mouse events are throttled to the monitor's frame rate though, which is in accordance to what UI events ask for, e.g. for mousemove:

The frequency rate of events while the pointing device is moved is implementation-, device-, and platform-specific, but multiple consecutive mousemove events SHOULD be fired for sustained pointer-device movement, rather than a single event for each instance of mouse movement. Implementations are encouraged to determine the optimal frequency rate to balance responsiveness with performance.

But then again, there is the pointerawupdate event that should not be throttled the same way. Probably having clearer hooks, or even more inclusion of UI events into HTML would help here.

Yes, I think several events are frame-throttled, and potentially differently at different browsers, and this is observable and unspecced (except for mousemove perhaps).

[smaug----]

FWIW, scroll/resize aren't UI events.
User input event handling scheduling is very much undefined currently. And things like pointerrawupdate make it even more complicated.
I believe in normal cases both Gecko and Blink try to handle non-pointerrawupdate (which might be still implemented in blink only) user input right before update rendering step. Some interesting scheduling issues rise when update rendering steps take too much time, or if there are tasks which take lots of time and lead to rending updates to be delayed.

WICG/scheduling-apis#67 has some information about the current behavior.

[noamr]

Finally got around to this, see #10007