Web Performance: Paint Timing

[mmocny]

Description

I’d like to propose a focus area for Interop 2023 focused on Web Performance: Paint Timing. This work would include:

• Expanding and clarifying existing spec text for marking time points related to rendering, in Paint Timing and Element Timing APIs (already in progress). This will help with interop in the face of fundamental differences in rendering pipelines and how animation frames are measured and reported across implementers.
• Clarifying which specific animation frames should report paint timings, especially when multiple paints are involved during an iterative rendering process, e.g. paints and sync/async image decode, paints during render throttling such as page visibility changes, etc.
• Creating new WPT tests which do a better job of accurately testing if reported timings represent the actual visual progress of loading / responsiveness as observed, e.g. first contentful paint

Rationale

First Contentful Paint (FCP) is one of the more important moments in Loading User Experience and is used by many Real User Monitoring (RUM) services to measure and improve the quality of UX.

Firefox, Safari, Chromium, and Edge all implement First Contentful Paint, and so it offers unique value to web developers using the Performance Timeline. However, the exact time values exposed have some significant differences, and these have been a persistent source of developer frustration, discussion within the Web Performance WG, and with a history of unresolved spec issues:

• Timestamps should be specified in an interoperative way w3c/paint-timing#62
• requestAnimationFrame callback and LCP renderTime should be the same w3c/largest-contentful-paint#104
• Expose LCP renderTime in non-TAO cross-origin images, when it doesn't reflect the image decoding time w3c/largest-contentful-paint#91

We believe this stems from two underspecified/undertested issues:

1. Which timings are exposed for any measured animation frame (there are multiple useful time points)
2. Which specific animation frames are measured

The goal of this Interop 2023 proposal is to focus on First Contentful Paint, and a focus on loading measurement issues specific to image decode modes and page visibility state. The improvements are expected to be foundational, and will apply to interop to Largest Contentful Paint, Element Timing, and Event Timing -- but implementing those measures on all platforms is not considered for this proposal.

Specification

• The HTML Standard: “Update the rendering” steps of the event loop
• Updating w3c Paint Timing specification
• Moving some parts of existing wicg Element Timing specification into HTML or paint timing.

Will limit the focus primarily on the details required for the first-contentful-paint entry.

(Some changes may also be made to related Largest Contentful Paint and Event Timing specifications, but only insofar as they are already defined in-terms-of Paint & Element Timing, and where this interop work will offer clarification and simplification in those areas).

Tests

Testing paint timings relies on accurate matching of visual loading experience, as observed by a user, with reported metrics.
Thus, Automated testing seems possible only for some types of tests (i.e., basic functionality, testing which animation frame is reported on, etc) but we may need to rely on Manual/Visual tests to ensure timing values are accurate and representative (in some sense, this is like testing the quality/timing of an animation).

• paint-timing directory
• fcp-only tests
• Will write new tests for FCP specific to image decode (sync, async, decode()) (Example test page used in the past)
• Will write new manual/visual tests to verify what is actually visually observed

Work may affect tests in other features using paint timings:

• Element Timing
• LCP
• Event Timing

[gsnedders]

This work would include: […]

This implies that you actually want an investigation effort, right, to improve the specs and tests?

[mmocny]

I was hoping to be able to improve the spec and tests within 2022 (it feels like we've resolved the major questions and just a bit of work... famous last words perhaps), but perhaps this could become and investigation effort if that proves to take longer.

(We may also want scope a larger effort if so.)

[gsnedders]

As https://github.com/web-platform-tests/interop/blob/main/2023/README.md#making-a-proposal says, a focus area is in principle primary about getting browsers to pass a set of tests for a feature, whereas an investigation effort is basically everything else. To be clear, this seems like a pretty well-scoped investigation effort with knowledge of what actually needs to be done, which in some ways really avoids the exploratory stages that one might consider the most investigative of most of the investigation projects.

[tbondwilkinson]

This would be useful for Google's server framework.

"We provide out of the box metrics for customers -- having those metrics be as useful as possible is a good goal"

[mmocny]

Thanks @tbondwilkinson for feedback.

@gsnedders After discussing with various stakeholders, I think that indeed it would be more appropriate to make this an investigation effort. From the necessary improvements to existing specs and tests, to the ambiguity of visual testing using WPT, and the potential/risk of increasing scope, etc.

I will update this issue with details of the investigation effort ASAP.

[foolip]

@mmocny as a reminder, today is the last day to refine proposals. What's missing here is an investigation roadmap, and #201 and #211 are good examples of what that could look like.

[mmocny]

Thanks for the examples.

Now that this proposal is changed to an investigation effort, I think this would be the Investigation Roadmap:

Investigation Roadmap

• Investigate and write up the options for interoperability testing of paint timing in WPT.
  • This may include some manual (visual) tests, but ideally mostly automated/ref tests.
• If required, do any infrastructure work required to enable timing sensitive testing of rendering during load.
• Build prototypes that showcase the above working across implementations.
• Propose spec changes that enable interoperable paint timing for animation frames (mostly the same as original description, above)
  • Expanding and clarifying existing spec text for marking time points related to rendering
  • Clarifying which specific animation frames should report paint timings
• Additionally, since this is no longer a focus area proposal, this investigation effort could now also include writing up how all related specs may be impacted by this work. Specifically: Largest Contentful Paint, Element Timing, and Event Timing.

Complete success would look like paint timing results being available on wpt.fyi by the end of Interop 2023. Working out how such results would be used as part of a future Interop metric is explicitly out of scope.

---

Also, Regarding "How widely used is this feature?" -- According to Chrome Use Counters, it has 32.3912% usage

[jensimmons]

@foolip We believe this is an Investigation Effort proposal, not a Focus Area proposal. But it's been marked Focus Area on the spreadsheet. Is that an error? Or does it reflect a lack of consensus?

[foolip]

@jensimmons it's an investigation effort, I just forgot to update the spreadsheet when it was changed. Fixing now...

[nairnandu]

Thank you for proposing Paint Timing for inclusion in Interop 2023.

We wanted to let you know that this proposal was not selected to be part of Interop this year. We believe this proposal is too broad, and that Interop 2023 is not the right venue to do this investigation. We encourage you to start a conversation with the relevant working groups to get any issues resolved.

For an overview of our process, see the proposal selection summary. Thank you again for contributing to Interop 2023!

Posted on behalf of the Interop team.