Specify the relationship between session visibilityState and window state

[toji]

A concern that we've talked about before which reemerged during #687 is specifying exactly how XRSessions should react to various changes in the browser state and, conversely, how the browser should react to sessions being displayed. This should be specced as normatively as possible to ensure consistent behavior between browsers.

For example: What happens to the window visibilityState and rAF loop when an immersive session starts on a single-display device (mobile or standalone)? What about on multi-display devices (desktop tethered headsets)? What happens to the XRSession when you minimize a window? What about when you switch windows? Switch tabs? Does the session end, does it's visibilityState change, or does it just keep running normally?

[billorr-google]

Are there already thoughts on this issue, and it just needs to be written up, or does it need thinking through?

For sensor-backed inline sessions, I think we should require frame focus to give out pose data (or match the requirements that sensor APIs have). I could see an argument for still calling raf callbacks when the frame is not focused.

For immersive sessions with external monitor, I think visibility in-headset should be sufficient to give out pose data and continue the session. A key reason that window focus shouldn't be required is that several VR systems will take focus to display their own UI. If someone minimizes the window while a session is occurring, the user in-headset doesn't know that and doesn't care. However, if the headset has a proximity sensor, and detects no user in-headset, we could save resources by bluring/hiding the session. Similarly, page visibility should report whether the 2d page is visible, not whether the vr page is visible in-headset.

For immersive sessions without an external monitor, I think the page is not visible while the session is visible. Window raf could be paused while the session is visible instead of the page.

If we're eventually going to add things like DOM overlay or DOM layers, it may make sense to keep window raf firing, and keep the page reporting as visible for immersive sessions.

[thetuvix]

Specifically, do we expect UAs to transition to a session.visibilityState of "hidden" when a user of a desktop VR peripheral takes off their headset?

Per the spec text, UAs may wire things up this way, since the headset's view at that point "cannot be seen by the user". This is how Chromium has currently wired things up on Windows MR headsets, for example. However, I would not be surprised if this differs across other XR backends in Chromium (and further, across other UAs).

@toji / @Manishearth: What is your understanding of the spec's intention for when a VR headset peripheral reports "hidden"? Does it include doffing the headset? If so, are we comfortable with apps taking an implicit dependency on this as the only API to tell you whether the headset is on or off? If not, should we add spec language to discourage UAs/platforms from implementing the API this way?

[Manishearth]

My personal model doesn't include taking the headset off, especially since many devices don't support that. I'm not strongly attached to this model, though.

It does feel like we should have consistency here.

[blairmacintyre]

I would ask the question "What is the anticipated impact on the application"? It's pretty easy to imagine applications (e.g., simulations with physics) entirely breaking if they go long periods with no rAF called, or pose data. It would not be uncommon to take off an HMD for a short period (even for things like "hand it to a friend to let them see what's happening") or long period (go to the bathroom).

On tethered systems, I may take the HMD off frequently, and may have the original window/tab completely hidden, as I interact with the desktop of the computer. A webXR app should not get degraded or suspended if the original tab is hidden, or if I take the HMD off to type something on my computer.

On the desktop, if I have a 3D thing in a tab, and the tab is hidden, it may stop getting fast updates, but (I think) it still gets updates slowly. Hidden tabs with audio/video input still continue to get input (I think ... I've hidden a tab with audio chat and still continued talking, and a little icon continues to indicate that the mic is working).

On an HMD, I suspect we'll be gated on the platform, to some extent: different platforms will make different choices about when to "put the device to sleep" (especially for stand alone HMDs) or suspend apps, if the device is taken off.

I do not think we should further limit things. So, my hope and expectation would be that if an HMD is removed, and the browser is the thing running, it keeps running until the platform puts things to sleep. This includes immersive sessions (and, I think, is most important for them). So, I think WebXR should keep going, sending full speed updates and rendering at full speed, when an HMD is removed, until the platform tells the app to go to do differently.

Put another way: I think an immersive WebXR session should be have similarly to a native app on the platform it is on, and I don't think we should dictate differently. User's expectations will likely be based on the platform.

[toji]

Trying to close this out.

Specifically, do we expect UAs to transition to a session.visibilityState of "hidden" when a user of a desktop VR peripheral takes off their headset?

If it can detect that state then yes, I would expect that. More specifically, in the majority of cases where a proximity sensor is present the device reaction to being doffed is to turn off the screen. Given that the session content isn't being shown at all that seems like a clear-cut `"hidden" situation. That said, I agree with Blair in that I want to allow for UA/platform leeway here, since there are circumstances where that may not be the most sensible approach.

Are we comfortable with apps taking an implicit dependency on this as the only API to tell you whether the headset is on or off?

Even on platforms that do report "hidden" when the headset is doffed I wouldn't expect that to be the exclusive circumstance under which it may be reported. And I have a hard time envisioning a developer course of action based on the assumption that "hidden" == doffed that would be egregiously bad if the hidden state is produced by viewing system UI instead.

As for the question of if visibilityState is tied to the window's visibilityState, I think that the answer is "no, but maybe implicitly because focus anyway." XRSession.visibilityState should answer one question and one question only: Is my content visible on the XR device? It's reasonable to expect various UAs and platforms to approach this differently, and implementations should trust what they report. We do have some existing, fairly strict requirements that say you can't get poses when the XRSession's document isn't focused, though, and it's implied that the window should be visible when focused. (The HTML spec allows obscured windows to still be considered visible for accessibility purposes, FYI.) As such if the page is reporting "hidden" then it doesn't have focus and the XRSession isn't getting poses. How the XR device handles stale content is platform-specific but it seems like in most headworn cases at least it's a pretty logical thing to hide the content for comfort purposes.

So the ultimate answer is, IMO, "no direct, spec enforced relationship." even though they may naturally end up correlated by platform behavior. @NellWaliczek and @Manishearth, any thoughts/concerns?

[Manishearth]

I agree with this position.

[NellWaliczek]

Sounds good to me.

[toji]

Given the agreement between editors, then, I'm closing this issue.