Support Buffered Haptic Effects

[kearwood]

The "pulse" function added in PR #32 provides an easy to use, direct way to provide a simple, momentary actuation of vibration motors in controllers.

The goal of buffered haptics to is increase the variety of effects that can be expressed. More advanced effects require millisecond-level precision updates to the actuators that provide haptic feedback. The underlying hardware API's allow submission of arrays of values that are often uploaded wirelessly to the hardware itself for high frequency execution.

I would like to provide the ability to run these haptic effects asynchronously. The UA can emulate the hardware buffering, so this functionality can be available on all controllers that support haptic feedback while taking advantage of hardware haptic buffering when available.

Fundamental features of the suggested interface:

• Synchronized playback of buffered haptics on multiple actuators
• One-shot playback of buffers
• Looped playback of buffers
• Attenuation of buffer playback
• Event on playback completion, enabling double-buffering of buffers streamed from Javascript
• Extensible with Javascript libraries to provide LFO (Low Frequency Oscillation), Envelopes (i.e. Attack, Decay, Sustain and Release), and integration with physics engines with attenuation of effects.

[kearwood]

The proposed WebIDL would fit within the Gamepad Extensions:

// The actuator type determines the force applied for a given
// "value" in GamepadHapticActuatorBuffer or GamepadHapticActuator
enum GamepadHapticActuatorType {
  // Vibration is a rumbling effect often implemented as an offset weight
  // driven on a rotational axis.
  // The "value" of a vibration force determines the frequency of the rumble
  // effect and is normalized between 0.0 and 1.0.  The neutral value is 0.0.
  "vibration",

  // Linear actuators shift a weight to an absolute position, changing the
  // center of gravity of the controller and resulting in an opposing force
  // that can be felt by the holder.
  // The "value" of a linear force is the absolute position of the actuator
  // relative to its center position and is normalized between -1.0 and 1.0.
  // The neutral value is 0.0.
  "linear",

  // XXX - Angular force (steering wheels), springs, dampeners, and other
  //       actuators should be defined as separate GamepadHapticActuatorType's
  //       here instead of being mapped to "vibration" or "linear".
};

// Each GamepadHapticActuator corresponds to a motor or other actuator that can
// apply a force for the purposes of haptic feedback.  In some cases, the
// physical arrangement of devices may be represented as a simpler model to this
// api.  For example, a device with multiple degrees of freedom may process
// the actuator values passed into the api with inverse kinematics to determine
// the position required for the physical actuators.
dictionary GamepadHapticActuator {
  // type determines the range and effect of the "value"
  // passed to pulse() and within the GamepadHapticActuatorBuffer.
  readonly attribute GamepadHapticActuatorType type;

  // Pulse applies a value to the actuator for duration milliseconds.
  // If any GamepadHapticEffect is playing, the value passed to pulse()
  // is applied additively, clamped to limits defined by the actuator type.
  // The returned promise will resolve true once the pulse has completed.
  Promise<void> pulse(float value, float duration);
};

// GamepadHapticBuffer represents a timeline of values to drive a
// GamepadActuator.
dictionary GamepadHapticBuffer {
  // actuator identifies the actuator that will be driven by the
  // effect.  If multiple GamepadHapticBuffer are driving the same
  // actuator, the values are combined additively and clamped within the
  // range defined by the actuator type.
  GamepadHapticActuator actuator;

  // It is recommended to normalize these values, as they can
  // be attenuated by setting GamepadHapticEffect.gain
  sequence<float> values = [ ];

  // Durations are expressed in milliseconds
  sequence<float> durations = [ ];

  // The number of iterations to repeat the set of values.
  // If iterations is 0, then repeat infinitely.
  attribute unsigned long iterations;
};

// GamepadHapticEffect describes any haptic effect that is more complex than
// a simple pulse.
interface GamepadHapticEffect {
  // buffers defines which actuators are being driven and with which pattern of
  // values.
  // A GamepadHapticEffect can affect a single or multiple Gamepads.
  // GamepadHapticEffect is associated to a Gamepad through
  // GamepadHapticBuffer.actuator
  // If multiple effects or pulses are playing simultaneously, the values of each
  // actuator are combined additively.
  attribute bool sequence<GamepadHapticBuffer> buffers;

  // When GamepadHapticEffect is playing, IsPlaying is true.
  // The GamepadHapticEffect does not begin playing until play() is called.
  readonly attribute bool IsPlaying;

  // play() starts playing the GamepadHapticEffect.
  // The returned promise will resolve true once the effect is completed or
  // stop() is called.
  // If any of the buffers have iterations set to 0 (infinite), the promise
  // will not resolve true until stop() is called.
  // Changes to the buffers are not reflected in a playing haptic effect and
  // will only be applied once the effect is played again.
  // If the haptic effect is already playing, play() will not interrupt
  // the haptic effect and will resolve fail the returned promise.
  // If you wish to play multiple of the same effect additively, multiple
  // GamepadHapticEffect's should be created, but can share GamepadHapticBuffer's.
  [Throws]
  Promise<void> play();

  // stop() ends playback of the effect and returns the values of the actuators
  // to the neutral value as defined by the actuator type.
  // It is not necessary to call stop() if all of the buffers have non-zero
  // iterations.
  void stop();

  // XXX Do we wish to have a pause() function?

  // gain is a 0.0 - 1.0 attenuation of HapticBuffer values.
  // Unlike changes to buffers, changes to gain are effective immediately on
  // a playing haptic effect.
  // Simple dynamic collision effects can be modelled with a set of short,
  // infinitely repeating HapticBuffer's and an ADSR envelope applied with the
  // gain attribute.
  attribute float gain; 
};

partial interface Gamepad {
  // hapticActuators enumerates haptic feedback actuators such as rumble motors.
  readonly attribute GamepadHapticActuator[] hapticActuators;

  // playingHapticEffects includes any GamepadHapticEffect that are playing
  // and driving any actuator in this Gamepad.
  readonly attribute GamepadHapticEffect[] playingHapticEffects;
};

[kearwood]

Based on initial feedback when this WebIDL was posted as a comment in PR #32, we may wish to replace "linear" with a more immediately relevant actuator type in GamepadHapticActuatorType.

Perhaps it would also be beneficial to either specify that this API could be used for force feedback or is limited to vibration based haptics.

[cvan]

Thanks for filing this. Also, want to make sure that #19 isn't neglected. I assume we want to keep things separate, even though pulse has been introduced to the Gamepad Extensions supports now. Just checking.

[luser]

This is ... quite a lot of API surface. One of the things we've always tried to do with the Gamepad API is keep the API surface small, so it's easy to use and reason about. I'm not convinced that the payoff is worth the complexity here. FWIW, I have previously rejected supporting force-feedback, since it doesn't seem to be in wide use in modern devices, but I don't know if VR hardware changes that. Modern gamepads just offer vibration motors of varying intensity, and I think we should expose that as a simple API.

[tkodw]

The functionality requested is mostly the same as what is specified in the Web Audio API. Linear actuators are strikingly similar to audio devices in the way that they receive data, and many would argue that they are low frequency audio devices. The Web Audio API could be extended to support linear actuators with a minimal increase in API surface area.

Is it a reasonable proposal to provide vibration through the Web Audio API?

[cvan]

Interesting idea. Is there precedent with the Web Audio APIs of interacting with or mapping relationships to peripherals. Beyond gamepads, can you think of other use cases that could justify extending Web Audio for vibration? (It's worth mentioning that this could also cause further confusion with the already existent Vibration API.)

[tkodw]

The Web Audio API has AudioContext.createMediaStreamDestination() intended for sending audio through the WebRTC API to play on a remote system or saving to a file. The same API can be used to send data to linear actuators built into a Gamepad. Support for something like the WiiU controller's screen and speaker could use MediaStreams in a similar way.

[kearwood]

Thanks for the feedback. Perhaps either a simpler API that allows javascript to keep a buffered stream of haptic actuator values filled is all we need. Everything else could be done in library space.

I'll also explore the possible media stream integration concept and give a couple of options for the webidl.