[css-view-transitions-1] Handle startVT for offscreen iframes

[khushalsagar]

It's unclear what should be done when an iframe is offscreen and rendering opportunities for it are throttled, see spec for that state here : "Remove from docs all Document objects for which the user agent believes that it's preferable to skip updating the rendering for other reasons".

It's unclear how ViewTransitions should be handled in this state. Suppressing rendering opportunities implies that the behaviour differs based on when that happens. If its before the old state was snapshotted, we'd hit a timeout and abort the transition. If its after the pseudo-DOM was setup, then its similar to the author starting CSS animations on an offscreen Document. It's probably better to have consistent behaviour here.

The decision for this should likely also influence scoped transitions? If the root element of that transition is offscreen for example.

[vmpstr]

Summarizing an internal sync:

• Relying on throttling seems unfortunate since that's an UA heuristic for the most part
• Relying on visibility is maybe a little better
• Ultimately, we think that we should actually prevent throttling if there is a pending capture view transition. This way, the view transition will always reliably start. After the capture is done, we can throttle and do the same thing as we do with other CSS animations: run them eventually

[khushalsagar]

After the capture is done, we can throttle and do the same thing as we do with other CSS animations: run them eventually

The downside here though is that we keep snapshots cached which is unnecessary memory for offscreen content. It also involves forcing rendering of offscreen content. #8282 could help with it but it's an opt-in. And it seems unfortunate to allow an iframe (potentially cross-origin) to slow down the main frame even when its offscreen because they didn't use this optimization.

My inclination here is to skip the transition if the iframe goes offscreen, similar to visibility. With some UA defined margin around the viewport to decide whether its offscreen in case scrolling brings it onscreen while the update callback is running.

[khushalsagar]

Summary for another conversation on this, it's between the follow options. "Current Behaviour" is least preferred but mentioning for completeness.

Force Rendering

If an offscreen iframe starts a ViewTransition force it to render, i.e., the UA is not allowed to throttle an iframe with a ViewTransition. We can relax it to say that this force mode is applicable until the pseudo-DOM is set up and animations (UA or author provided) have started.

Because once the animations have been setup, it's no different from a regular CSS or web animation on the page. The animation progresses based on its timeline an the UA can optimize rendering similar to other offscreen animations. When the animations finish, all the setup will be torn down.

The pro for this is it better aligns with how CSS and web animations work on the platform today, its not web observable whether the content is offscreen. And avoids edge cases where the iframe comes onscreen while the updateCallback is running, resulting in a flash of intermediate DOM state.

The con is its suboptimal. Regular CSS or web animations don't force the browser to render offscreen content, while this will because the old DOM state will be torn down. So it consumes compute and memory for content never shown to the user. It's also a footgun, both the main and iframe might not realize that transitions in offscreen iframes are costly.

Skip transition

If the iframe is offscreen (based on a UA defined margin), transitions in it are skipped. The behaviour is the same as the Document being hidden, spec'd here and here.

The pro/cons for this is inverse of "Force Rendering". Its more optimal but can result in flashing of intermediate DOM states if the iframe is scrolled offscreen. For example, an abrupt change in scroll position using scrollbars.

Current Behaviour

The behaviour in the spec today is that after script calls startVT, the VT object is created and it waits for a rendering lifecycle update to cache the next frame. If the iframe is throttled, that rendering update won't happen until its un-throttled. So the iframe's VT would be blocked from starting until then.

The pro for this is that it takes care of both: don't use resources for offscreen content and don't flash intermediate DOM states. But it defers updating the DOM so the user unnecessarily sees stale content when the iframe comes onscreen. It's also possible the author is deferring some work until the transition's finished promise resolves, which is also delayed.

@nt1m @emilio fyi.

[css-meeting-bot]

The CSS Working Group just discussed [css-view-transitions-1] Handle startVT for offscreen iframes.

The full IRC log of that discussion

<TabAtkins> Khush, ping me with any difficulties or for review
<fantasai> khush: optimization that if an iframe is off-screen, we don't run lifecycle updates or slow them down
<fantasai> khush: So as an optimization, offscreen doesn't get animated
<fantasai> khush: During view transitions, looking for that refresh
<fantasai> khush: but for iframe won't refresh until iframe comes on screen
<fantasai> khush: given it's already off-screen, user isn't going to see any animation
<fantasai> khush: so suggestion that if it's offscreen, treat it as invisible
<fantasai> khush: or if iframe is detached from tree
<fantasai> khush: current spec is to skip the transition
<fantasai> khush: just update the DOM directly, since user won't see the animation anyway
<fantasai> khush: Question here is, what if iframe is not detached, but is far enough off-screen that it's not being animated by the browser
<emilio> q+
<fantasai> khush: currently what happens is transition once you bring it on screen
<fantasai> khush: other option is to skip the transition
<fantasai> khush: third option is to force the update and do the transition
<astearns> ack emilio
<fantasai> khush: once the browser has set up the animation, just like other animations the UA won't need ot run the animation, just update the timeline, as long as it's offscreen
<fantasai> emilio: I'm not a fan of forced-rendering, because it allows [missed]
<fantasai> emilio: you can call requestAnimationFrame and then startViewTransition
<fantasai> emilio: I tend to think current behavior is more consistent with regular animation frame would work
<vmpstr> q+
<fantasai> emilio: so I think current behavior is fine
<fantasai> emilio: skipping transition allows you to detect this more directly, I don't think it's great
<fantasai> emilio: we rely on the page not knowing whether it's throttled
<fantasai> khush: because iframe is not ??, can't they already detect offscreen?
<fantasai> emilio: well yeah, but you don't know if it's because of slow machine or because offscreen
<vmpstr> s/??/getting rAF callbacks/
<fantasai> emilio: can act somewhat differently, but iframe isn't allowed to unthrottle itself, which is what forced rendering would do
<astearns> ack vmpstr
<fantasai> ??: We'll timeout by the time the animation is on screen, so if call startViewTransition callback won't run until at least one rAF callback, and by that time we are likely to skip the transition
<astearns> s/??/vmpstr/
<fantasai> khush: timeout starts after we cache the old state. Just happens to ???
<fantasai> vmpstr: doesn't seem great
<fantasai> emilio: Should we change that so that the timeout starts when you actually call the API?
<fantasai> khush: don't see any reason not to
<khush> q+
<fantasai> astearns: other questions/comments?
<fantasai> emilio: Given we unthrottle not exactly when you're in the viewport bounds, but in some range, likely that by the time you scroll to the iframe transition has finished anyway
<fantasai> stephenfromgoogle: Is throttling behavior thought to be interoperable at this time? Vague in the spec.
<fantasai> stephenfromgoogle: would we introduce an interop issue if we don't force rendering?
<vmpstr> +1
<fantasai> flackr: that was one of my concerns about not forcing rendering. it would expose more of those differences in behavior
<fantasai> emilio: that difference is already exposed by rAF tagging
<astearns> s/tagging/timing/
<fantasai> emilio: I think it's somewhat reasonably interoperable, at least I assume all browsers to be throttling visibility:hidden and very far off screen
<fantasai> emilio: this is an area that could get improvement, especially now that lazyloading thresholds are a thing
<flackr> q+
<fantasai> stephenfromgoogle: I can tell you, our throttling code in Chromium has no margin. If an iframe is even 1px outside viewport, it will be throttled. That's different form lazyloading, which does have a margin.
<fantasai> emilio: worth discussing in an HTML spec issue
<fantasai> emilio: in any case, I still think forced rendering is not a great option
<fantasai> emilio: if you call startVT, then you can force engine to unthrottle you all the time, which is not great
<astearns> ack khush
<fantasai> khush: +1 everything Emilio said
<fantasai> khush: especially for one frame
<vmpstr> q+
<fantasai> khush: with other CSS animations, unless animating we don't need to spend resources on it, but here do
<fantasai> khush: so would prefer current behavior
<fantasai> khush: Yes, different browsers use different margins, and different features have different margins, being exposed doesn't seem like a big constraint
<astearns> ack flackr
<fantasai> flackr: difference with animations is that you can always produce the thing that would have been drawn
<fantasai> flackr: whereas with VT, once you draw the updated, you have destroyed the old state
<fantasai> flackr: can't recover if it suddently becomes visible
<fantasai> flackr: One use case was starting a view transition on a frame, and then bring it into view
<fantasai> flackr: maybe there's an alternative to consider who's starting the transition
<fantasai> flackr: if starting within hidden frame, ... but if starting from root frame, could unskip frame by moving it into view?
<astearns> ack vmpstr
<fantasai> vmpstr: whether we keep these frames alive or not is also a function of whether started when iframe on screen and then moved offscreen or started offscreen
<fantasai> vmpstr: Also don't understand concern wrt frame throttling. You could set timeouts and force rendering in a script
<fantasai> vmpstr: why would they use VT to unthrottle?
<fantasai> flackr: Browsers can force minimum timeouts on setTimeout
<fantasai> vmpstr: it would force it to run rAF?
<fantasai> vmpstr: iframe would do more work?
<fantasai> flackr: yes, it would force it right away, but setTimeout would wait a second
<fantasai> vmpstr: what's the attack vector here?
<fantasai> emilio: I'm not sure whether there's a good way... I've seen frames consuming too much CPU unintentionally
<fantasai> emilio: I can't think of a recent case where they do it intentionally
<fantasai> dholbert: maybe an ad wants to start playing immediately rather than skipping a few frames, so continuously burns CPU in the background so it's ready
<fantasai> khush: use case I'm concerned is, it's hard for an iframe to do the right thing without a lot of code
<fantasai> khush: whereas with other animations, just browser doesn't spend resources. This is the first case where we would force rendering
<fantasai> vmpstr: more wondering about second case
<fantasai> vmpstr: we could enforce setTimeout limits
<fantasai> vmpstr: concerned that you call your startViewTransition for some effect, and that will only start running when you're on screen
<emilio> q+
<fantasai> vmpstr: so I would force rendering or skip trnasition, but not doing thing until on screen
<astearns> ack emilio
<fantasai> emilio: alternatively set timeout mechanism to start when you call API?
<fantasai> vmpstr: [missed]
<flackr> q+
<fantasai> vmpstr: much better than current behavior
<astearns> ack flackr
<fantasai> flackr: one possibility, when we start the animation we could start it with a negative start delay, so that it would be timed as if it had started on time
<vmpstr> s/[missed]/it's timing dependent -- whether you've scrolled the frame into view within the timeout or not/
<fantasai> flackr: this would basically defer the work until the animation became visible
<fantasai> khush: you'd have to render to cache?
<fantasai> flackr: when it becomes visible. If it doesn't become visible until end of duration, then you can skip it
<fantasai> khush: complexity of all the animation after caching old DOM, you still have to do at least one render to cache it
<fantasai> flackr: or you defer the dom change
<fantasai> khush: which pseudo-elements get rendered depends on the new DOM
<fantasai> khush: we don't know until we run the callback
<fantasai> flackr: you defer until it becomes visible
<fantasai> flackr: when it becomes visible, you do pre-snapshot, you do the update, you determine animations, and you determine that the animations are finished, so immediately jump to end state
<fantasai> khush: sounds like skip
<fantasai> flackr: except it fires all the events
<fantasai> flackr: and [missed]
<fantasai> flackr: it's only "skipped" ...
<fantasai> khush: what's the goal? we don't want the author to know we skipped it?
<fantasai> flackr: there's no difference between whether your frame was on screen or offscreen
<fantasai> flackr: if it comes on-screen during animation, you'll see the animation
<fantasai> flackr: I haven't fully thought this out, but thinking that if you know when the request to start the animation occurred, that's the start of your timeline
<fantasai> khush: like current behavior, except animations will finish faster
<fantasai> emilio: might also be consistent with how time-based CSS animations work
<fantasai> emilio: if you have an iframe, and then scroll down, by the time you scroll up again the browser has done no rendering updates, but animation jumps to the correct point
<fantasai> emilio: that's worth consideration
<fantasai> vmpstr: capture right now can take some amount of time, and so that time would then be immediatley skipped into the animation, which won't be a smooth experience
<fantasai> vmpstr: essentially the animation start waits until the capture is available; with this change it would be a frame or two of delay
<khush> q+
<fantasai> vmpstr: so those first 2 frames would be immediately skipped into the third frame of the animation, which is not ideal
<fantasai> flackr: that's a fair concern, possible ways to mitigate that
<fantasai> flackr: you take the time until the frame started as your negative start delay
<fantasai> vmpstr: worth considering this model, but need to be careful about the math
<astearns> ack khush
<fantasai> khush: my issue is this approach is complicated. What benefit do we derive from it?
<fantasai> khush: ...
<fantasai> khush: what's the benefit of bringing iframe on screen with the old DOM?
<emilio> q+
<fantasai> vmpstr: it supports cases of, suppose your animation is 30s and starts offscreen, and then you scroll it into view partway through
<fantasai> vmpstr: it looks like everything was working all along
<fantasai> khush: you eventually want user to see the new state
<fantasai> khush: point of doing animation is to go from old state to new state nicely
<fantasai> astearns: maybe we should look at the new option and bring it back
<astearns> ack emilio
<fantasai> emilio: perhaps more realistic use case for this model is, consider a lazyloaded iframe
<fantasai> emilio: page being loaded has a view transition while loading
<fantasai> emilio: right now, given stephen's comment about no margin
<fantasai> emilio: lazyloading kicks in first, page is invisible, then you call startViewTransition. If you skip it, then you discard everything the author wanted even though user is about to scroll the iframe
<flackr> +1
<fantasai> emilio: That model doesn't seme too complicated, just need to handle the first two frame jump
<fantasai> emilio: It behaves more consistently with our APIs also
<fantasai> emilio: rAF we just delay your callback, this seems similar
<fantasai> emilio: I think there are more use cases that justify this kind of model
<astearns> ack fantasai
<fantasai> fantasai: basically agree with astearns and emilio
<fantasai> fantasai: let's explore flackr's model, and come back
<fantasai> vmpstr: ok, let's look into the issue and comment with our findings

[khushalsagar]

@flackr interesting question post the resolution, how should we handle web animations? Especially since they can be chained. The author might use the finished promise of 1 web animation to start another web animation. Do we keep adding the delta for how long the iframe was offscreen before the transition started to all animations associated with that transition.

[flackr]

If they're chaining promises, I think it would be up to them. This is no different then if you chain promises on regular web animations for content in a far away frame that they'll only start at the point where the lifecycle updates to trigger the finish promise and it would be up to the developer to apply a negative start delay if they want to start the next animation when the previous one finished from a time perspective.

[khushalsagar]

This is no different then if you chain promises on regular web animations for content in a far away frame that they'll only start at the point where the lifecycle updates to trigger the finish promise

So for regular web animations, we decide the start time at the time the author calls element.animate. But if the animation reaches its end time while its still offscreen, we won't fire any events for it until it comes onscreen..? Is that consistent across browsers? I have a memory of this coming up in person with @emilio.

[emilio]

I think so since animation events are fired from the rendering update code right?

[emilio]

Btw, the side track of trying to improve interop here with iframe throttling is whatwg/html#10333

[flackr]

I think so since animation events are fired from the rendering update code right?

The timelines section of the web-animations-1 spec covers animation events, in particular:

When asked to update animations and send events for a Document doc at timestamp now, run these steps:
...
4. Let events to dispatch be a copy of doc’s pending animation event queue.
5. Clear doc’s pending animation event queue.
...
7. Dispatch each of the events in events to dispatch at their corresponding target using the order established in the previous step.

So yep, they are fired as a result of the "update the rendering" steps.

[khushalsagar]

Cool. So seems like for parity with general animations, we would delay the startTime of the animations targeting pseudo-elements in the first update-the-rendering loop after the pseudo-elements are generated. Post that, all animations (CSS transitions, animations or web animations) get their start time as usual.

[vmpstr]

There's also the matter of the update callback time here. So, the current events we have are:

t0: startViewTransition is called
t1: capture is requested**
t2: capture completes
t3: callback starts***
t4: callback ends
t5: pseudos are created and animation is ready to start

** this includes the request being unblocked to propagate wherever it needs to, not just pending
*** t2 and t3 are generally the same time, iirc

At t5 right now, the animation progress time is 0. The proposal is to set the progress to something like (t5-t0) so the animation conceptually starts when startViewTransition is called. However, we also discussed excluding some times like the capture turnaround (t2-t1). I think we likely also want to exclude callback runtime (t4-t3)?

So, would then should the initial progress of the animation at t5 be (t1-t0)? This is reasonable for css animations. However, for web animations that start at the ready promise resolution (t5), it seems a little awkward to start that animation with non-zero progress and presumably only the animations that target the vt pseudos. This makes coordinating multiple animations difficult, imho.