Behavior when XRSession.visibility is 'visible-blurred'

[NellWaliczek]

[Disclaimer: This issue is one of several being filed to capture discussions that began either on #638, on #689, or at the most recent F2F]

The current spec text states

XRSessions MUST have their visibility state set to hidden or visible-blurred when the user is interacting with potentially sensitive UI from the user agent (such as entering a URL) in the trusted environment.

It also states

A state of visible-blurred indicates that imagery rendered by the XRSession may be seen by the user, but is not the primary focus. requestAnimationFrame() callbacks MAY be throttled. Input is not processed by the XRSession.

What other, if any, concrete requirements should be added to the behavior of the session RAF, getViewerPose(), getPose(), and the data these APIs return while in this state? For example, should callbacks be forced to throttle down to, uh I don't know, 10fps? Should pose data be quantized or rounded? PR #638 touches on this topic a bit, but doesn't seem to provide any concrete guidance.

[toji]

There does appear to be some relevant text in #638: Under the section about "Mandatory conditions for exposing XRPose data", one of the conditions is:

Currently focused area belongs to a document whose origin is same origin-domain with the origin of the given active document.

This could probably stand to read more clearly regarding trusted UI, but I would imagine that if the trusted UI has focus then the document itself does not? This is already the case with traditional pages, where if the URL bar for a page is focused then document.hasFocus() returns false.

Deriving a suggestion that visible-blurred receives no poses from the above text contradicts the current spec text and my personal hopes for the feature. Seems like the reasoning behind it is laid out in the privacy and security repo, where it's listed as a guard against input sniffing:

In theory, limiting pose input frequency may reduce a site's ability to infer gaze, input data, location, or perform user profiling. In practice however, such throttling would certainly affect the user experience and it is unlikely to provide additional security.

For example, even at 1Hz, the user's location and gaze information could likely be inferred. Similarly, while touch input data snooping was performed at frequencies as low as 20Hz, it is not known whether there is in practice a lower bound that prevents such detection. In either case display frame rates are likely to be higher than 20Hz and thus pose data at a higher frequency would be needed to maintain a reasonable user experience.

I don't think the above would rule out delivering periodic poses to produce, for example, cube maps of the scene that could be reprojected behind the UI? Also, it's not clear to me if this should apply to something like the Steam UI being shown. The natural behavior of that scenario is the "blur" the running application by graying it out, rendering at a lower resolution, and suspending input tracking, but it still appears to track the user's head motion in realtime. A strict reading of the text above would imply that we should suspend poses the the page entirely when any overlay UI like that is displayed because the document is no longer explicitly in focus, which is an unfortunate experience hit in those scenarios. (For example: Desktop browsers don't black out when you focus on a different window, but then again desktop browsers also generally don't have access to your head pose...)

[joshmarinacci]

I think head-pose as input should be distinguished from controller as input. The key here is that whatever will be used to interact with the trusted UI should not be presented to the blurred window. So unless you need to accept a dialog by using your head pose as a gaze proxy (like in a cardboard style device) then I think we can safely provide head pose to a blurred window.

[NellWaliczek]

I wonder if even under those circumstances there isn't still a threat associated with head pose? Like... maybe people naturally move their head to follow the motion controller motion? And if so, it seems like a password might be sniffable?

[johnpallett]

The considerations in this issue seem related to conditions for when iframes are allowed to access pose data... some of the threat vectors (input sniffing, gaze tracking) and possible mitigations are similar. @NellWaliczek I couldn't find an issue for that specific question, do you want to track it here?

[NellWaliczek]

We definitely need to make sure we're tracking any potential threats. Can you elaborate a bit on the overlap you see with this issue? Based on that, we can figure out if filing a separate issue would be the better course of action.

[johnpallett]

For background, as @toji mentions above, the design proposed in #638 requires that pose data only be available to a session that is visible, and same-origin to, the currently focused area:

Visibility state of the document is "visible".
Currently focused area belongs to a document whose origin is same origin-domain with the origin of the given active document.

This requirement was intended to cover threat vectors for two scenarios:

• 'visible-blurred' (where the user agent is presenting sensitive UX) and
• the behavior of x-origin iframes when they are out of focus or not visible.

Input sniffing is a threat vector in both scenarios, whether the input is to a user agent, or to a web page on a different origin than the XR session. As an example of the latter, a top-level domain might present an input for sensitive data (e.g. a password field) that does not trigger a 'visible-blurred' scenario, but might allow an x-origin iframe to sniff the input.

Gaze tracking is another relevant treat vector... arguably less of a concern for 'visible-blurred', but it's relevant when considering x-origin iframes that may wish to learn about user behavior on the top-level domain.

The exploration above might be useful in both scenarios, and could address both threat vectors. For example, the idea of rendering a cube map might be a valid approach in both scenarios.

I don't have an opinion as to whether there should be multiple issues, but there does seem to be similarities in both the threat vectors and possible mitigations.

[toji]

Been giving this quite a bit of thought, and I think that there's a bit of a disconnect between the title of this issue and the thing that we're all discussing.

visible-blurred, along with the other XRVisibilityState values, are not intended to enforce any particular privacy or security policy. They are intended to merely communicate the state of the application without commenting on why, precisely, the application is in that state. It would actually be problematic, IMO, if developers found that they could infer that the visible-blurred state always meant "The user is doing something they don't want me to see." Instead, that state should very simply communicate any time the content is still visible but not in the "foreground". Nothing more or less. As has been pointed out previously, bringing up some element of the SteamVR UI is a pretty good example of this in action.

Based on that, the "Behavior when XRSession.visibility is visible-blurred" follows pretty naturally: You don't have access to input (because something else has focus), and you may not get frames processes at full speed. The reason for frame throttling, if any is applied, is ambiguous and left to the discretion of the UA. The application should then use that information to respond in contextually relevant ways, such as pausing media.

Perfect example: If you ever take focus away from Beat Saber for any reason when you come back the song is paused and you have to click a button to resume, after which you're given a very brief countdown and the gameplay starts up again.

Now, that doesn't really answer the questions that everyone is actually discussing in this thread, which is more accurately stated as "When is it safe to use visible-blurred while interacting with sensitive UI." I'm going to assert at this point that doing so is a quality-of-life improvement for users rather than a necessity, and one that won't be applicable on all devices. (Mobile devices don't have the horsepower to run both a foreground and background scene, so they hide the content altogether.) It's also a scenario that can be allowed additively, once we've have more time to further explore what we feel the guidelines for operating in that mode should be. As such, I recommend that we change the spec, for the June milestone, to indicate that interacting with sensitive UI must set the visibilityState to hidden until further notice.

For discussion of when we can re-enable visible-blurred in those situations I think a new issue (not in the June milestone) is appropriate, and for larger discussion of when each XRVisibilityState should be applied I think #696 is the right place.

[toji]

PR #742 removes the ability to use visible-blurred with sensitive UI for the time being, as described above.

[toji]

New issue files specifically to discuss the mitigations necessary to allow use of visible-blurred with sensitive UI: #743