Skip to content
This repository has been archived by the owner on Apr 4, 2018. It is now read-only.

Linking between VR web pages #22

Closed
brianchirls opened this issue Mar 17, 2016 · 19 comments
Closed

Linking between VR web pages #22

brianchirls opened this issue Mar 17, 2016 · 19 comments

Comments

@brianchirls
Copy link

I seem to recall some discussion of enabling linking directly from one VR web site to another without having to stop presenting. The spec as it is doesn't seem to handle that.

This is a potentially tricky case, so I understand wanting to hold off on implementation until more progress is made in the field. But has there been any discussion about what this might look like in an API? Are we confident that the API as it currently stands leaves room for that in the future?

@AlbertoElias
Copy link

From what I understood, once there has been a user interaction to start presenting in the HMD, you can move to different VR capable sites without having to take the headset off.

@brianchirls
Copy link
Author

So how does the destination web page know that it's already presenting when it starts running?

And what happens if that destination web page doesn't know how to handle VR?

@cvan
Copy link
Contributor

cvan commented Mar 24, 2016

So how does the destination web page know that it's already presenting when it starts running?

And what happens if that destination web page doesn't know how to handle VR?

Last summer, we prototyped a standalone VR backwards-compatible browser based on Firefox's Graphene project (which is Desktop B2G, similar to Electron). You have an app manifest, an index.html, and your scripts which have access to special APIs (which you can use obviously to create your own application – or browser! – in HTML+CSS+JS). Check out the MDN docs, if you're interested. It's quite similar to Chrome's Content API for its webviews.

Check out this short video demoing our Horizon prototype browser.

À la the meta[name=viewport] tag, we added a meta[name=viewmode] tag (see code in Firefox here). This is really only useful for Graphene-based apps which can listen for mozbrowser-specific events such as mozbrowsermetachange.

Unlike the viewport tag, viewmode does not block. This meant that we wouldn't know exactly when a page loaded whether it would be a full VR experience, a 2D classic/mono experience, a 2D experience that also has a button to enter VR mode, a 2D-to-3D VR responsive experience, a 2D+3D+mobile+desktop responsive experience, etc. It gets complicated.

And you want to know which type of experience it is before you render the next page. Do you project a non-VR site as a 2D plane? That's what we did with Horizon. That isn't doable in content with pure WebGL in Chrome, however.

We looked into the Navigation Transitions spec (good critique blog post here) or writing an extension or chrome-level code in Firefox that would allow a kind of hijacking/pausing of the navigation. Navigation Transitions isn't fully spec'd or implemented in Chrome nor Firefox, so that seemed premature to count on that.

Chrome versions 31-38 used to rely on <meta name="mobile-web-app-capable" content="yes"> to identify whether a web site was a "web app" that could be added to the homescreen. (As I'm sure you know, Safari for iOS have their own Apple-specific meta tags, which was the inspiration for this mobile-web-app-capable one.

Anyway, Chrome for Android 39 stopped using the mobile-web-app-capable meta tag. Instead, "web apps" are defined as sites with a manifest and a Service Worker.

That's when I proposed using the Web App Manifest for VR-specific metadata. It seems like a good idea because you can reference a single manifest for all documents on your server (e.g., suppose you have a VR world that contains a different document per level). Huge plus would be not having to propose and invent a new <meta> tag for each new WebVR feature we want to support.

Example: Michael Blix at Samsung had proposed a meta[name="vr-background"] (à la [meta[name="theme-color"]]). This would be a perfect case for using a key in the manifest.

Since the Web App Manifest spec has a non-normative section suggesting to use proprietary keys, the browser could use that information to identify the viewmode of the site and then figure out the projection. Or if a 2D page is encountered, the user agent could decide to open that in a new tab and display a notification message in the headset. It's really up to the user agent to decide how to handle "legacy" content. But parts of this probably ought to be spec'd, at the very least as non-normative text in the spec.

The Manifest spec is explicit that manifests should be fetched asynchronously by the browser. So obviously fetching and reading the manifest metadata would slower than synchronously parsing a meta tag.

If you're interested in some of the pros/cons of the approaches the Mozilla VR was doing WebVR prototyping and prepping for the WebVR API overhaul for v1.0, check out this Google Doc.

And, heh, I just found a short summary I wrote a few months ago that mostly summarises what I said above.

Anyway, we stopped working on our Horizon browser prototype because of a bunch of technical limitations:

  • we were not using pure WebGL but an experimental Firefox branch of CSS-VR for the HUD and projecting planes (CSS-VR turns out was too slow - and not to mention scary and risky because we were overloading CSS properties)
  • backwards compatibility was costing us a lot of time (Firefox didn't support spatial navigation at the time; so we were going off some unimplemented W3C spec and writing our own solution using HMD gaze-based cursor + gamepad controls for clicking and tabbing between focussable elements)
  • we had to propose a lot of Chrome-level changes to support VR experience using the mozbrowser API, and we couldn't at the time justify that these changes would amount to an experience we wanted; another option was to fork Firefox and strip out the UI ourselves, but maintaining a fork is costly and risky

We should loop in @caseyyee from the Mozilla VR team who has done a lot of thinking with me on the browser and link traversal UX prototyping. Also, we should loop in @mkeblx from Samsung. The Samsung Internet Browser for Gear VR has to handle all of these things in the very near future, as they're looking to support first-class legacy 2D and WebVR experiences.

@kearwood
Copy link

This probably belongs as non-normative text, as the behavior can vary from one browser to another without affecting the WebVR API, but I can share our design related to link traversal and permissions:

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

  • If there are no VRDisplays connected, the returned promise resolves without any user interaction or display, to enable sites to detect WebVR capabilities without adding friction for users without VR hardware.
  • If the site is already in WebVR or is identified as a VR site linked from another VR site, the returned promise resolves without any user interaction.
  • If the site is not already in WebVR, wasn't linked from a VR site, and there is at least one VRDisplay to return, the browser prompts the user for permission to enter VR. The promise does not resolve until the user accepts or declines the request. If the user declines, the promise is rejected and the site can not access the VRDisplay objects or use their attributes in an exploit to track the user. The user may also decide to "accept always" for the site.

@brianchirls
Copy link
Author

If the site is already in WebVR or is identified as a VR site linked from another VR site, the returned promise resolves without any user interaction.

I assume in this case it resolves with the devices, right?

I'm still not clear on how this solves the problem. Assuming my site has been linked from another VR site and is already presenting, how do I know that so I can start rendering in stereo? When that promise resolves very quickly with a list of devices, how can I tell the difference between being already in VR and the user having selected "accept always"?

@kearwood
Copy link

I assume in this case it resolves with the devices, right?
Exactly

I'm still not clear on how this solves the problem. Assuming my site has been linked from another VR site and is already presenting, how do I know that so I can start rendering in stereo? When that promise resolves very quickly with a list of devices, how can I tell the difference between being already in VR and the user having selected "accept always"?

Would a flag living along side VRDisplay.IsPresenting and VRDisplay.IsConnected help here? I suppose it would need to be something that returns "true" in this case.

@brianchirls
Copy link
Author

Yes, I imagine something like that would work. I'd want to try it out to be
sure, of course. There are always unexpected quirks on different devices
and scenarios.
On Mar 24, 2016 3:33 PM, "Kearwood Gilbert" [email protected]
wrote:

I assume in this case it resolves with the devices, right?
Exactly

I'm still not clear on how this solves the problem. Assuming my site has
been linked from another VR site and is already presenting, how do I know
that so I can start rendering in stereo? When that promise resolves very
quickly with a list of devices, how can I tell the difference between being
already in VR and the user having selected "accept always"?

Would a flag living along side VRDisplay.IsPresenting and
VRDisplay.IsConnected help here? I suppose it would need to be something
that returns "true" in this case.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#22 (comment)

@mkeblx
Copy link
Contributor

mkeblx commented Apr 1, 2016

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

Can you clarify thoughts in regards to prompt display in case of third bullet point above. Any site calling getVRDisplays to query WebVR capabilities would show dialog right at page load potentially, even if no immediate intent to present/enter fullscreen. Is there a good way for site being able to determine if any vrdisplay connected (without getting data access) without calling getVRDisplays and having prompt displayed? Would ideally like to avoid that for just that one piece of info.

@cvan
Copy link
Contributor

cvan commented Apr 1, 2016

@mkeblx à la gamepadconnected and gamepaddisconnected for navigator.getGamepads, vrdisplayconnected and vrdisplaydisconnected events will be emitted without your having to call navigator.getVRDisplays to query for the display. I never actually spec'd this but I have an issue on file to: #15 (comment)

should definitely spec that soon.

@cvan
Copy link
Contributor

cvan commented Apr 1, 2016

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

I don't understand why we need UX prompts for entering VR. and calling navigator.getVRDisplays doesn't necessarily mean the user is going to enter VR. calling VRDisplay.requestPrompt() does though.

IMO, we should talk to @marcoscaceres, et. al. about adding WebVR to the Permissions API enum so we can use that API.

and until the Permissions API is supported, in the meantime, follow the same security model used by Pop-ups, Pointer Lock, Fullscreen API, etc. - that is, require an "engagement gesture" on the first entering of VR:

see https://html.spec.whatwg.org/#allowed-to-show-a-popup

@mkeblx
Copy link
Contributor

mkeblx commented Apr 1, 2016

Great, forgot about those, that'd work. Will comment on issue for specing out.

I think permission prompt on getVRDisplays more to do with tracking data a la geolocation API, etc. so does make sense there in cases where will return a VRDisplay. But only when actually at point of requesting that access (or ahead of time with Permissions API). Or should we view pose data as same as same as device orientation data w.r.t. permissions?

Or are there two different permissions: data access, and ability to present/go fullscreen?

@cvan
Copy link
Contributor

cvan commented Apr 1, 2016

I think permission prompt on getVRDisplays more to do with tracking data a la geolocation API

in Chrome, Geolocation permissions (since 43) have been controlled by the Permissions API. see https://developers.google.com/web/updates/2015/04/permissions-api-for-the-web and https://googlechrome.github.io/samples/permissions/

Firefox implementation isn't done yet, but we should plan on using the Permissions API regardless.

But only when actually at point of requesting that access (or ahead of time with Permissions API). Or should we view pose data as same as same as device orientation data w.r.t. permissions?

FWIW, push and midi already have specific keys. I think we should do the same for vr: add specific keys for queryDisplays, allowRequestPrompt, etc. (proper naming TBD, obviously).

thoughts?

@cvan
Copy link
Contributor

cvan commented Apr 1, 2016

full pseudo-code example would go something like this:

var canQueryDisplays = false;
var canRequestPresent = false;  // Can enter VR based on a user-initiated gesture (e.g., `click`).
var canRequestPresentOnLoad = false;  // Can enter VR immediately on document load (see `NOTE` below).
var vrDisplay;

/*

// If you want to test for the permissions explicitly …

navigator.permissions.query({name: 'vr', queryDisplays: true}).then(function (result) {
  canQueryDisplays = this.state === 'granted';
  result.onchange = function () {
    canQueryDisplays = this.state === 'granted';
  };
});

navigator.permissions.query({name: 'vr', requestPresent: true}).then(function (result) {
  canRequestPresent = this.state === 'granted';
  result.onchange = function () {
    canRequestPresent = this.state === 'granted';
  };
});

// NOTE: We may not ever want this to be a toggleable permission.
navigator.permissions.query({name: 'vr', requestPresentOnLoad: true}).then(function (result) {
  canRequestPresentOnLoad = this.state === 'granted';
  result.onchange = function () {
    canRequestPresentOnLoad = this.state === 'granted';
  };
});

*/

function successQueryDisplays (displays) {
  console.log('`navigator.getVRDisplays` access granted');

  if (displays) {
    vrDisplay = displays[0];
    console.log('VR display found and using:', vrDisplay);
  }

  return vrDisplay;
}

function errorQueryDisplays (err) {
  console.warn('`navigator.getVRDisplays` was blocked for this reason: %s', err);
}


function requestPresent (vrDisplay) {
  return vrDisplay.requestPresent({source: webglCanvas});
}

function errorRequestPresent (err) {
  console.warn('`VRDisplay.requestPresent` was blocked for this reason: %s', err);
}


function successEnterVR () {
  enterVRButton.style.display = 'none';
  console.log('`VRDisplay.requestPresent` access granted and entered VR on document load:', vrDisplay);
}

function errorEnterVR (err) {
  enterVRButton.style.display = 'block';
  console.log('Could not enter VR for this reason: %s', err);
}


window.addEventListener('vrdisplaypresentchange', function () {
  if (vrDisplay && vrDisplay.isPresenting) {
    enterVRButton.style.display = 'none';
  } else {
    enterVRButton.style.display = 'block';
  }
});


// Query for a VR display.
navigator.getVRDisplays().then(successQueryDisplays, errorQueryDisplays)
  // Try to enter VR immediately if we can.
  .then(requestPresent, errorRequestPresent)
  .then(successEnterVR, errorEnterVR);


enterVRButton.addEventListener('click', function () {
  if (!vrDisplay) {
    console.warn('Cannot enter VR because no displays found');
    return;
  }

  if (vrDisplay.isPresenting) {
    console.warn('Cannot reenter VR because already presenting');
    return;
  }

  requestPresent(vrDisplay).then(successEnterVR, errorRequestPresent).catch(errorEnterVR);
});

@mkeblx
Copy link
Contributor

mkeblx commented Apr 1, 2016

Permission API makes sense, plus pseudo code LGTM.

@mkeblx
Copy link
Contributor

mkeblx commented Apr 7, 2016

@kearwood: If the user declines, the promise is rejected and the site can not access the VRDisplay objects or use their attributes in an exploit to track the user.

@cvan: First, do we ever plan on adding a permission to call navigator.getVRDisplays()? I'd prefer no, at least for this first version.

Is queryDisplays permission required? One one hand it definitely is exploitable tracking data, on other hand so is mouse position data. Also lets page determine if and what model of device you own. You can also imagine other exploits when system/chome UI is showing.

Obviously it would simplify things if not required.

@mkeblx
Copy link
Contributor

mkeblx commented Apr 8, 2016

Also currently spec Security Concerns says:

Content does not need to request user permission to present to the VR HMD

This could be clarified unless intend no permission required (just user gesture required perhaps). I kind of don't like having two permissions. (And anyway if did would probably have a UA setting to ignore for all sites for one/both).

@cvan
Copy link
Contributor

cvan commented Apr 8, 2016

I vote no to introducing a permission to query the displays (i.e., to call navigator.getVRDisplays()). If we need one later, we can add one (and have it use the Permissions API).

This could be clarified unless intend no permission required (just user gesture required perhaps).

That was my initial suggestion. Introducing a permission (in the Permissions API sense) seems a bit much, introduces a dependency on the Permissions API (which only Chrome has shipped to release), and introduces a ton of user experience implications that I don't think we've properly thought through with link traversal. Once we add it, it's going to be next to impossible to remove.

Per my earlier comment, I vote for this simpler approach to gate on a user gesture for the initial user request of entering VR:

follow the same security model used by Pop-ups, Pointer Lock, Fullscreen API, etc. - that is, require an "engagement gesture" on the first entering of VR:

see https://html.spec.whatwg.org/#allowed-to-show-a-popup

@toji
Copy link
Contributor

toji commented Apr 8, 2016

FWIW I also vote no on introducing a permission here, but ultimately that will be up to the security/privacy people on the Chrome team and not me. :( In the meantime I won't add one voluntarily and I'll push back if one is requested.

@cvan
Copy link
Contributor

cvan commented Jun 2, 2016

Moved to immersive-web/webxr#30

@cvan cvan closed this as completed Jun 2, 2016
@MozillaReality MozillaReality locked and limited conversation to collaborators Jun 2, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants