F2F Followup: Frame of Reference Proposal

[toji]

During the face to face meeting in Kirkland there was a lot of discussion around the current Frame of Reference/Coordinate System propsal by Nell (#149). I heard some feedback around the API shape from web platform folks, and concerns about the complexity of the current proposal, as well as some confusion surrounding the names. This is my attempt to simplify it a bit and apply the recommendations I've received around the API shape. Would love feedback on this! (Posting as a separate issue from Nell's proposal for easier reading. Lots of comments on the pull request.)

Issues being addressed:

• Update API shape
• Reduce feature complexity

Overview:

I suggest reducing the proposed pair of types (FrameOfReference and CoordinateSystem) down to a single type. Which of those terms we move forward I don't have a preference for, I'm using FrameOfReference here but would be equally happy with CoordinateSystem. This prevents some confusion about which object does what, and removes a layer of abstraction that doesn't provide a clear benefit for the initial API release.

I also propose that we reduce the initial set of FrameOfReference's from three (Attached, Stationary, Stage) to two with hopefully more descriptive names. (Relative and Room). The RelativeFrameOfReference, which provides poses relative to the displays initial readings, can be requested to provide either 3DoF or 6DoF data, and can fall back to 3DoF even when 6DoF is requested. The RoomFrameOfReference is the basic standing space that Nell's proposal described, with the origin at floor level, simply with a different name (Room is hopefully more intuitive than the Stage terminology we've used to date.)

Some potentially useful functionality like getTransformTo is dropped, since that will be easy to add on in a future version, and it would be interesting to hear from developers if it's functionality that the need first. I've also left off the room sizes for the moment. I'd definitely like to see them included, but probably as a more general boundary geometry that probably merits more discussion outside this particular issue.

Finally, following web platform feedback, the API has moved from create*FrameOfReference functions to using new *FrameOfReference. Constructors are allowed to throw if the VRDisplay doesn't support that type. (This sounded weird to me, but apparently it's the webby thing to do.) In order to prevent forcing devs to feature detect via try/catch a way to return the supported frame's of reference is also provided.

Rough IDL:

(Note - VRPoseProvider is used here as a deliberately vauge term instead of VRDisplay because a related proposal may shift where this functionality lives. See #179. You can read it for the moment as VRDisplay.)

interface VRFrameOfReference {};

dictionary VRRelativeFrameOfReferenceOptions {
  boolean position = true;
};

// This one can't throw since it must always be supported.
[Constructor(VRPoseProvider poseProvider,
             optional VRRelativeFrameOfReferenceOptions options)]
interface VRRelativeFrameOfReference : VRFrameOfReference {
  boolean trackedPosition;
};

[Constructor(VRPoseProvider poseProvider),
 Throws]
interface VRRoomFrameOfReference : VRFrameOfReference {
};

partial interface VRPoseProvider {
  sequence<DOMString> supportedFramesOfReference();
  // Yes, we still need to decide return vs. populate.
  VRFrameData getFrameData(VRFrameOfReference frameOfReference);
};

Example usage:

// Create a 6DoF relative frame of reference
let sixFrameOfReference = new VRRelativeFrameOfReference(vrDisplay);
if (!sixFrameOfReference.trackedPosition) {
  // Requested 6DoF but got back 3DoF. System may provide neck modeling.
}

// Create an explicity 3DoF relative frame of reference
let threeFrameOfReference = new VRRelativeFrameOfReference(vrDisplay,
                                                           { position: false });

let roomFrameOfReference = null;
// Check for room scale support
if (vrDisplay.supportedFramesOfReference().includes('VRRoomFrameOfReference')) {
  // Create a room scale (implicitly 6DoF) frame of reference
  roomFrameOfReference = new VRRoomFrameOfReference(vrDisplay);
}

// Query the frame data with one of the frames of reference.
let frameData = vrDisplay.getFrameData(sixFrameOfReference);

Oustanding questions:

• I feel like specifying { position: false } of a VRRelativeFrameOfReference should explicitly disallow neck modeling, which would be the desired output for things like photospheres/360 videos. Not sure if all platforms can support that, though, or if there's cases where it's not the desired effect.
• Not sure if supportedFramesOfReference should return a sequence of enums. If so, what should they be? Or maybe there's a more elegant way to handle that feature test entirely?

[judax]

I like the simplification of the API a lot to be honest. I recall that during the meeting there were some concerns about using room scale as a method to describe the use case and proposed: "eye" and "floor" as possible reference points. I have to admit that I like these better even though they might not be the facto standard as "room". Another possibility could be to use "relative" and "absolute" frame of references as I understand that one is relative to the initial eye position/orientation and the other relative to an absolute origin (floor).

VRRelativeFrameOfReference
VRAbsoluteFrameOfReference

It could be ideal that @NellWaliczek could review this proposal and see if it still fits the needs of their use cases (Windows Holographics). I could understand that the coordinate system might be an interesting additional value to be provided.

[toji]

I would really like to make some forward progress on these spec changes, so I think I'm generally going to interpret lack of feedback over the past two weeks as "Nothing negative to say". I'll start working this into the explainer, and put together a pull request in the next day or so assuming I don't hear any protests.

Keep in mind that everything is pretty malleable at this point, so this should be viewed as a starting point rather than an immutable part of the spec.

[toji]

One more update as I'm collecting these proposals into a new explainer. After sketching out a few examples of how these work I've got a simpler proposal still:

dictionary VRFrameOfReferenceInit {
  attribute boolean position = true;
  attribute boolean floorRelative = false;
};

interface VRFrameOfReference {
  readonly attribute boolean position;
  readonly attribute boolean floorRelative;
};

partial interface VRSession {
  VRFrameOfReference? getFrameOfReference(VRFrameOfReferenceInit options);
};

The basic pattern with the dictionary is one that borrows from WebGL's context creation attributes. Specifically: The dictionary you pass in indicates properties you would like to have, and the object that gets passed back sets the values to what's actually supported. A couple of examples:

// On an Oculus Rift
getFrameOfReference({ position: true }) => VRFRameOfReference{ position: true, floorRelative: false };
getFrameOfReference({ position: false }) => VRFRameOfReference{ position: false, floorRelative: false }; // Forces 3DoF, probably no neck model
getFrameOfReference({ floorRelative: true }) => VRFRameOfReference{ position: true, floorRelative: true }; // Room scale or Standing space, Floor at Y = 0

// On a Daydream View
getFrameOfReference({ position: true }) => VRFRameOfReference{ position: false, floorRelative: false }; // Asked for 6DoF, only got 3DoF. Maybe still provides neck model.
getFrameOfReference({ floorRelative: true }) => null OR VRFRameOfReference{ position: false, floorRelative: false }; // Not sure yet what makes the most sense here.

I know that previously I indicated that the change to use constructors was informed by input from web platform folks, but after giving it a lot of thought I think that a function to retrieve an instance like this may be a better pattern long term. In particular, if we extend this system to something like Gamepad we'd need a way to get a frame of reference for gamepads that can track poses but don't have an associated VRDisplay. You'd have to overload the constructor for each type of object that may need to create a frame of reference, or have them all inherit from some base "PoseProvider" object like I had stubbed in at first. Both valid options, and I'm happy to consider either, but this pattern eliminates a need for a separate function to check if the frame of reference type is supported (you just try to create one and get null back) and since the multiple frame of reference types have been boiled down to one with a dictionary of options there's no concern about potentially unbounded function growth. (A previously cited issue with the create pattern in the original proposal.)

[toji]

There's been a side email conversation between myself, Microsoft, and Oculus on this topic so I wanted to update this issue with some of what's been discussed there:

• The VRFrameOfReferenceInit seems too complex for the current needs and can contain some possibly nonsensical combinations (Does anyone really want { position: false, floorRelative: true }?) so a simple enum may serve us better.
• Nell from Microsoft still feels that the concept of a Coordinate System that's not explicitly tied to a Frame of Reference is valuable, but suggested that it could be a base class for the Frame of Reference as a way of simplifying usage.
• While I stripped them out of my proposal MS and Oculus both are in favor of keeping methods for getting play area bounds and the transforms between two coordinate systems.

(Hope I didn't miss or misinterpret any of that.)

So that leaves us with IDL that looks more like this:

partial interface VRSession {
  // Feels like these should be promises? I'm willing to go either way on it, though.
  Promise<VRFrameOfReference> createFrameOfReference(VRFrameOfReferenceType type);
  Promise<VRPlayAreaBounds> getPlayAreaBounds(VRCoordinateSystem coordinateSystem);
  VRFrameData? getFrameData(VRCoordinateSystem coordinateSystem);
};

interface VRCoordinateSystem {
  // This is one of those areas where it feels like using DOMMatrix would make a lot of sense.
  Float32Array? getTransformTo(VRCoordinateSystem other);
};

enum VRFrameOfReferenceType {
  "HeadModel", // Debating if this one is necessary
  "EyeLevel",
  "FloorLevel",
};

interface VRFrameOfReference : VRCoordinateSystem {
  readonly attribute VRFrameOfReferenceType type;
};

interface VRPlayAreaBounds {
  readonly attribute float minX;
  readonly attribute float maxX;
  readonly attribute float minZ;
  readonly attribute float maxZ;
};

[toji]

Closing this issue since it seems like we've come to a rough consensus, which is now reflected in the explainer. See:

• https://github.com/w3c/webvr/blob/master/explainer.md#room-scale-tracking-and-boundaries
• https://github.com/w3c/webvr/blob/master/explainer.md#appendix-b-proposed-idl

One change worth noting: I've gone back to using the "Stage" verbiage after some conversations with @NellWaliczek. The concept of "Room" isn't bad, but a room may have multiple floor elevations and may extend well past the trackable bounds. As such it seems useful to have an explicit concept of a "bounded, floor relative play space" and WebVR 1.1's use of the word Stage seemed to fit the bill.

If anyone has issues with the functionality as outlined in the explainer please open a new issue so we don't have to sort through so much conversation history.