Added a supplemental "Extensions" spec

[toji]

This is to provide a place to maintain the specs for the proposed additions to the Gamepad API for reference, discussion, and bug fixing while the main spec is in the process of progressing to a CR.

This is maybe a bit unorthodox, but while I understand the desire to not push a bunch of new changes into the spec before it progresses to CR the lack of a formal spec for these proposals is turning into a blocker for me. Hopefully this strikes a happy balance that allows those who are interested to reference this spec when needed and have discussions around the features it contains while those who are only interested in the main spec can blissfully ignore it.

[sgraham]

supported

[sgraham]

I think it's fine to merge this in as an indication of direction for experimentation.

I hope that the experimental part of the API would only be shipped behind some sort of manual opt-in so as not to cause reliance on the details of one particular browser's implementation until we can have more discussions on the design tradeoffs.

[toji]

Fixed the typo. :)

I'm definitely sensitive to not wanting to get locked into a spec without wider discussion about it. (That's another big reason why I wanted this spec to be available: To facilitate conversations about the features.) At the moment Chrome is only planning to make them available behind a flag.

[cvan]

can you fix casing to "GitHub"?

[toji]

Thanks for the feedback, @cvan. Addressed all of your comments.

[kearwood]

In able to achieve better haptic texturing, perhaps we could expose a haptic feedback buffer, similar to the one exposed in the Oculus SDK:

https://developer3.oculus.com/doc/1.7.0-libovr/structovr_haptics_buffer.html

Such a buffer would include vibration samples for multiple GamepadHaptics so they can be precisely timed together. For devices that do not have a buffer playback implementation in the underlying API, this could be emulated by the UA with the benefit of working asynchronously and with precise timing. Similar to the Oculus SDK model, a function to get the state of the asynchronously playing haptic buffer would also be required. The processing model should additionally describe what happens when a new buffer is submitted before the prior buffer has completed.

I think it's still useful to have a simple function like vibrate() that emits a single pulse; however, could we rename it to something that wouldn't be confused with a later added function for submitting a buffer? Perhaps "pulse()" instead of "vibrate()"?

[kearwood]

In order to support devices such as leap motion and perception neuron, perhaps Gamepad could be extended to enumerate child Gamepad objects in a tree hierarchy similar to the one abstracted by OSVR. A string or enumerated value could differentiate the children based on their relation to the parent. The child Gamepad objects would not be enumerated independently of their parents and could only be owned by a single parent.

Reason for establishing the relationship within the Gamepad rather than the GamepadPose is that some buttons and input axis would be associated with a particular articulated part of the controller, such as a glove or suit with fingertip sensors.

[kearwood]

The children Gamepad enumeration could be added in a later spec revision; however, this may have an influence on the GamepadHand enumeration. Perhaps GamepadHand could be renamed in a manner that reflects the relationship of the Gamepad with it's parent. With no parent, it could be simply identifying the "left" or "right" hand. With a parent, it could identify the particular joint "finger", "elbow", etc..

I propose that we rename "GamepadHand" and "Gamepad.hand" to something without "Hand" in their names. Perhaps "GamepadRelation" and "Gamepad.path"?

Finally, by associating the Gamepad with the VR Display's DisplayID, we could map OSVR's contept of a path tree to WebVR:

https://osvr.github.io/presentations/20150419-osvr-software-framework-path-tree.pdf

[toji]

Regarding devices like Leap Motion and Perception Neuron: I don't believe that these are good fits for the Gamepad API. Other VR input devices like Oculus Touch, The Vive wands, and the Daydream controller all resemble a more traditional game controller with motion tracking capabilities. They have buttons and axes, etc. Something like Leap Motion has effectively zero mapping to the base Gamepad API and only a tenuous mapping to these extensions. As such I don't think it's wise to target them with this API.

That being said the Leap Motion does have a pretty nice Javascript API that's driven by WebSockets. It allows the API to be very targeted and expressive, and I haven't found it to be terribly latent. It's installed with their runtime, so it's available on any PC which has their drivers. I've found that to be a very effective model for exposing that sort of hardware, and if we get to a point where that level of hand tracking is normalized and looking for a standardized web API I think we would do well to take notes from their existing work in that space..

[toji]

I was unaware that touch had haptic feedback at all, so thanks for pointing out the function! The base vibration function this one is based on takes an array of durations that they refer to as the vibration pattern. It's a series of on-off timespans. So [500, 100, 1000, 100, 500] means:

• Vibrate for 500ms
• Turn off for 100ms
• Vibrate for 1s
• Turn off for 100ms
• Vibrate for 500ms

That seems similar-ish to the Touch haptics, though I don't really understand the timing of that API. Looks like you provide a bunch of amplitudes and it just ticks through them at a constant rate? In any case, for our haptics needs we'd probably want duration and intensity pairs, which might look a little ugly. Probably look something like:

gamepad.haptics[0].vibrate([1.0, 0.5, 0, 1.0], [300, 500, 1000, 200]);

Which would be:

• Full intensity for 300ms
• Half intensity for 500ms
• No vibration for 1s
• Full intensity for 200ms

:P

[kearwood]

I think duration and intensity pairs could easily be mapped to what ever an underlying API requires. For example, if a haptic effect requires a buffer with a fixed timestep, the UA could "bake" a buffer using the duration-intensity pairs passed in. In cases where the underlying hardware only accepts discrete updates of intensity, the UA could implement the loop that updates the intensity and run it asynchronously with more precise timing than possible in Javascript.

Note that in many cases, the controllers integrate a microcontroller that performs this inner loop. (Some motor driver IC's themselves or the integrate such a microcontroller for this purpose). Having this offloaded to the controllers when possible would eliminate any artifacts that are caused by latency and wireless connections.

For controllers with multiple haptic motors that work in unison, would instantiating an effect with vibrate() on each motor provide sufficient timing accuracy?

[kearwood]

Related to the naming of "vibrate". Would we want to also support haptics based on linear actuators or rotational torque through the same interface?

An example of a linear actuator is Apple's taptic engine. In this case, instead of speed of rotation, you could apply a specific position to a weight to shift its center of gravity precisely. Motion simulator chairs may also fall into this category.

An example of a rotational torque would be steering wheels. In this case, a particular amount of torque is desired to pull the wheel away from its current position.

Linear actuators and rotational torque haptics may not be desired now, but perhaps we could ensure that names used in WebIDL and normative text can be generic enough that it won't make for awkward mappings in the future. (eg. Use value instead of intensity, perhaps).

[kearwood]

I would like your opinion on this syntax:

gamepad.HapticEffect([0, 1], [1.0, 0.5, 0, 1.0], [1.0, 0.5, 1.0, 1.0], [300.0, 500.0, 1000.0, 200.0]);

• First array is list of haptics
• Second array is list of "values" for haptic[0]
• Third array is list of "values" for haptic[1]
• Fourth array is duration between samples, shared between all haptic motors in the effect

gamepad.HapticEffectCompleted(function() { ..... });

• When the haptic effect has completed, the callback function is triggered

gamepad.HapticEffect([0, 1], [0.0], [0.0], [0.0]);

• Triggering a new haptic effect cancels the last if still playing.
• This call effectively sets haptic[0] and haptic[1] values to 0, stopping any effects.

Alternately, would a dictionary with named members be better for the parameters?

[kearwood]

Agree about the devices like leap motion and perception neuron. Perhaps if they are implemented in a separate API with a tree-like structure, each node could reference interfaces exposed separately in the gamepad api. If so, we could effectively ignore those for now.

[kearwood]

I have given this a bit of thought and iterated on some WebIDL for an interface that makes the easy "pulse" effects simple while enabling high frequency buffered haptic effects.

Here is a rough draft of proposed WebIDL:

// The actuator type determines the force applied for a given
// "value" in GamepadHapticActuatorBuffer or GamepadHapticActuator
enum GamepadHapticActuatorType {
  // Vibration is a rumbling effect often implemented as an offset weight
  // driven on a rotational axis.
  // The "value" of a vibration force determines the frequency of the rumble
  // effect and is normalized between 0.0 and 1.0.  The neutral value is 0.0.
  "vibration",

  // Linear actuators shift a weight to an absolute position, changing the
  // center of gravity of the controller and resulting in an opposing force
  // that can be felt by the holder.
  // The "value" of a linear force is the absolute position of the actuator
  // relative to its center position and is normalized between -1.0 and 1.0.
  // The neutral value is 0.0.
  "linear",

  // XXX - Angular force (steering wheels), springs, dampeners, and other
  //       actuators should be defined as separate GamepadHapticActuatorType's
  //       here instead of being mapped to "vibration" or "linear".
};

// Each GamepadHapticActuator corresponds to a motor or other actuator that can
// apply a force for the purposes of haptic feedback.  In some cases, the
// physical arrangement of devices may be represented as a simpler model to this
// api.  For example, a device with multiple degrees of freedom may process
// the actuator values passed into the api with inverse kinematics to determine
// the position required for the physical actuators.
dictionary GamepadHapticActuator {
  // type determines the range and effect of the "value"
  // passed to pulse() and within the GamepadHapticActuatorBuffer.
  readonly attribute GamepadHapticActuatorType type;

  // Pulse applies a value to the actuator for duration milliseconds.
  // If any GamepadHapticEffect is playing, the value passed to pulse()
  // is applied additively, clamped to limits defined by the actuator type.
  // The returned promise will resolve true once the pulse has completed.
  Promise<void> pulse(float value, float duration);
};

// GamepadHapticBuffer represents a timeline of values to drive a
// GamepadActuator.
dictionary GamepadHapticBuffer {
  // actuator identifies the actuator that will be driven by the
  // effect.  If multiple GamepadHapticBuffer are driving the same
  // actuator, the values are combined additively and clamped within the
  // range defined by the actuator type.
  GamepadHapticActuator actuator;

  // It is recommended to normalize these values, as they can
  // be attenuated by setting GamepadHapticEffect.gain
  sequence<float> values = [ ];

  // Durations are expressed in milliseconds
  sequence<float> durations = [ ];

  // The number of iterations to repeat the set of values.
  // If iterations is 0, then repeat infinitely.
  attribute unsigned long iterations;
};

// GamepadHapticEffect describes any haptic effect that is more complex than
// a simple pulse.
interface GamepadHapticEffect {
  // buffers defines which actuators are being driven and with which pattern of
  // values.
  // A GamepadHapticEffect can affect a single or multiple Gamepads.
  // GamepadHapticEffect is associated to a Gamepad through
  // GamepadHapticBuffer.actuator
  // If multiple effects or pulses are playing simultaneously, the values of each
  // actuator are combined additively.
  attribute bool sequence<GamepadHapticBuffer> buffers;

  // When GamepadHapticEffect is playing, IsPlaying is true.
  // The GamepadHapticEffect does not begin playing until play() is called.
  readonly attribute bool IsPlaying;

  // play() starts playing the GamepadHapticEffect.
  // The returned promise will resolve true once the effect is completed or
  // stop() is called.
  // If any of the buffers have iterations set to 0 (infinite), the promise
  // will not resolve true until stop() is called.
  // Changes to the buffers are not reflected in a playing haptic effect and
  // will only be applied once the effect is played again.
  // If the haptic effect is already playing, play() will not interrupt
  // the haptic effect and will resolve fail the returned promise.
  // If you wish to play multiple of the same effect additively, multiple
  // GamepadHapticEffect's should be created, but can share GamepadHapticBuffer's.
  [Throws]
  Promise<void> play();

  // stop() ends playback of the effect and returns the values of the actuators
  // to the neutral value as defined by the actuator type.
  // It is not necessary to call stop() if all of the buffers have non-zero
  // iterations.
  void stop();

  // XXX Do we wish to have a pause() function?

  // gain is a 0.0 - 1.0 attenuation of HapticBuffer values.
  // Unlike changes to buffers, changes to gain are effective immediately on
  // a playing haptic effect.
  // Simple dynamic collision effects can be modelled with a set of short,
  // infinitely repeating HapticBuffer's and an ADSR envelope applied with the
  // gain attribute.
  attribute float gain; 
};

partial interface Gamepad {
  // hapticActuators enumerates haptic feedback actuators such as rumble motors.
  readonly attribute GamepadHapticActuator[] hapticActuators;

  // playingHapticEffects includes any GamepadHapticEffect that are playing
  // and driving any actuator in this Gamepad.
  readonly attribute GamepadHapticEffect[] playingHapticEffects;
};

[kearwood]

Example usage of a simple pulse haptic:

// Vibrate with an amplitude of 0.7 for 25 ms:
gamepad.haptics[0].pulse(0.7, 25.0);

With a bit more validation:

gamepad.haptics.forEach(function(haptic) {
  if(haptic.type == "vibration") {
    haptic.pulse(0.7, 25.0);
  }
});

[kearwood]

If you think this is in the right direction, I would be glad to write up some examples of physically modeled haptic effects that could be implemented with the rest of the proposed interfaces.

[toji]

That's a really interesting proposal! Still absorbing it all, but one question comes to mind immediately: Do you know of any gamepads that use linear actuators? A brief search yields a lot of motors but no gamepads. I'm guessing a lot of the old Force Feedback joysticks used linear actuators, but I consider that API to be pretty well dead and buried. I know you pointed out the taptic engine as an example, but I don't think that we'd be targeting that with the gamepad API.

Sounds like a cool idea, but I'm reluctant to pursue it unless there's devices out there that can make good use of it (and that we can test against.)

[kearwood]

For linear actuators, that may include devices such as this:

http://www.geomagic.com/en/products-landing-pages/haptic

Such a device would expose 3 axis aligned actuators which drive the physical motors through inverse kinematics.

Perhaps "linear" may be confused with LRA (Linear Resonant Actuator) like those in the steam controller. LRA's are capable of oscillating at specific resonant frequencies only and should be represented as "vibration" actuators.

Fundamental to this API is the concept of additive combining of haptics and force feedback systems. My intent was to show that the API is agnostic of actuator type and extensible by showing at least one additional type of actuator that has an absolute value with a range centered around 0.

Perhaps a more immediate use case to illustrate the use of an absolute position actuator would be to describe the rotational friction, rotational spring (centering), and rotational force actuators as used in a steering wheel controller.

[kearwood]

@toji Perhaps we could implement the WebIDL as I had listed, but with only the "vibration" GamepadHapticActuatorType. This should solve our immediate needs in supporting the peripherals bundled with the currently shipping VR platforms. Content would be encouraged to validate that the actuators are "vibration" type for forward compatibility with more advanced haptic and force feedback devices as they are added.

[kearwood]

As mentioned in Issue 24, it would be ideal to expose enough haptic feedback functionality to enable libraries to implement an SDL2 compatible API.

[toji]

I think I'm pretty good with that. Can we perhaps even further scaled back for the immediate moment and start with just:

enum GamepadHapticActuatorType {
  "vibration",
};

interface GamepadHapticActuator {
  readonly attribute GamepadHapticActuatorType type;
  Promise<void> pulse(double value, double duration);
};

And work up from there? I like the buffered playback idea but it's a fair amount of extra complexity and probably warrants further discussion. In the meantime I'd really really like to get this spec published and linkable quickly. Fortunately the way you have the API segmented means that it can be added in later without any compatibility issues.

[toji]

Okay, updated the spec so that it at least covers the GamepadHapticActuator, GamepadHapticActuatorType, pulse, and pulls a bunch of verbiage over from the comments in the proposal @kearwood made. I'd definitely like to see that side of things expand, but this feels like a good starting point!

[kearwood]

This looks good to me. This won't prevent us from adding the rest later after careful review and won't delay immediate benefit

[sgraham]

I'm going to merge this as a checkpoint. It's not fixing anything in stone of course, but it's easy to take further feedback and make more changes with a base to start from.

[cvan]

@toji @kearwood does one of you want to file a follow-up PR or issue for those interface changes?

[kearwood]

@cvan I would be glad to make a follow-up issue to continue the discussion of the buffered haptics that was skipped in this PR.

[luser]

Yeah, let's take individual new features to new PRs to keep things easy to follow. I think this is a sensible approach--we've been stalled on getting the existing spec through the process but there's definitely a desire to add things to support VR use cases etc, so it'll be good to have a place to collect those and discuss them while we finish off the v1 spec.