Sensor discovery

[tobie]

Define how sensors are discoverable by the developer (especially what happens when you have multiple sensors of the same type).

Proposed resolution: There must be a way for developers to find out which sensors are available to a resource, which ones become available during the page's lifetime, and which ones get disconnected. The precise API by which this must be exposed is dependent on the research carried out in #9 to determine how sensor register themselves with a browsing context. Such an API must be designed to allow developers to gracefully handle sensors misrepresenting themselves either accidentally or on purpose.

[tobie]

/cc @richtr as that's one of the concerns you brought up during the call.

[richtr]

As general feedback on this: what a device reports itself as supporting vs what a device actually supports are quite often different things. e.g. see w3c/deviceorientation#12.

Put another way, you can only really be certain as a developer that a specific sensor is supported by trying to access that sensor's data directly. Sensors can also (theoretically) come and go during the lifetime of a web page so checking for support for a device sensor through e.g. the existence of an Interface being attached to the window or navigator object does not provide the level of granularity we probably want in sensor detection.

[tobie]

@richtr agreed. That's my main concern re feature detection. This is more about enumerating multiple sensors of the same type. The approach taken by getUserMedia (which you suggested in #8 (comment)), has issues. For example, the Amazon fire phone has 6 cameras, while the getUserMedia spec only seems to allow for four different cameras (front, back, right and left sides).

[tobie]

Tying this in to #9. I think it's critical here to have a better understanding of how sensors are discovered/report themselves to the browser and which information is available about them when navigating to a new resource and which information needs to be queried asynchronously.

Obviously, sensor unplugging or plugging during page lifetime also needs to be handled.

Finally, there remains the issue that a device can misrepresent itself, either accidentally or on purpose, and that must be accounted for in the APIs.

Proposing that we resolve this issue like so:

There must be a way for developers to find out which sensors are available to a resource, which ones become available during the page's lifetime, and which ones get disconnected. The precise API by which this must be exposed is dependent on the research carried out in #9 to determine how sensor register themselves with a browsing context. Such an API must be designed to allow developers to gracefully handle sensors misrepresenting themselves either accidentally or on purpose.

[rektide]

Some existing examples of specs tackling discoverability:

• Gamepad API enumerates gamepads with getGamepads() and exposes gamepadconnected and gamepaddisconnected events.
• Media Capture and Streams has an enumerateDevices() that returns a promise of all devices, and exposes a devicechange event signalling any change, with no payload.

The enumeration returning a Promise (as Media Capture does) seems safer for implementors. As a consumer, I'm much prefer getting connected/disconnected events to payload-less change events: I can imagine rather sizable sensor configurations, such that diffing all the sensors each time someone comes or goes would be un-fun/possibly slow.

One possibility is that the enumeration/event only contain a copy of the SensorOptions, since the Sensor itself might involve initializing the peripheral. Having to initialize all devices to look for the one you want sounds like a bad, dangerous idea.

[tobie]

One possibility is that the enumeration/event only contain a copy of the SensorOptions, since the Sensor itself might involve initializing the peripheral.

I've added a start method in prevision of this. Constructing the sensor object itself should have no side effect.

[tobie]

Our current goal is to make sure the design of the Sensor object API, which we'll be shipping first, doesn't clause doors for the discoverability API (which we'll tackle later) while still providing value right now.

@rektide, if you spot areas where we're failing to do that, please flag these asap.

[anssiko]

We might need something similar for the WebVR API, see immersive-web/webxr#185 (comment)

[pozdnyakov]

We might not need to add discovery API per se at the moment, but simply having a "location" (or similar) parameter in SensorOptions would be beneficial.

[tobie]

@pozdnyakov see: https://w3c.github.io/sensors/#identifying-parameters

[larsgk]

(moved from #178 a case and some concerns related to discovery):

A use case that might very well happen soon:

I am sitting in my house looking at a temperature widget.

1. The web app widget is connected to both my phone temperature sensor and a temperature sensor outside my kitchen window.
2. I walk out to my car, losing connection with the sensor in my house - but keeping connection with the phone's sensor.
3. I start my car and my web app widget discovers the inside and outside temperature sensors in the car and shows them in the widget

now... there are a few issues with the current API for this to happen:

1. to have a new TemperatureSensor() in the first place, the browser needs to know about a 'TemperatureSensor' or it needs to be added with some polyfill (= not very generic)
2. I don't see any mechanism/events for adding/removing sensors
3. I see only a short example of a very specific way of addressing multiple sensors in the tire pressure example ... IMO, we need sensor discovery based on physical properties (e.g. tire pressure sensors can (also) be found by searching for sensors related to 'rotating motion' & 'air pressure' ... )

[larsgk]

I am in the process of adding support for different sensors and actuators via WebBluetooth and WebUSB, which would definitely not be "few and static" => requiring some add/remove/discover mechanism.

Has this effort stranded?

In my case, I may have 0 - many of each type of sensor (accelerometer, gyro, temperature, etc. ) and actuator (RGB LED, vibra, etc.) and I'd prefer if I could make it within ght Generic Sensor API (+ my own 'reverse' Generic Actuator API)

[anssiko]

This effort has definitely not stranded. It was just put to the Level 2 feature bucket while the group is focusing on Level 1 issues that affect the Chrome implementation, see https://lists.w3.org/Archives/Public/public-device-apis/2017Jun/0019.html

@larsgk, would you like to champion this issue? A first good step would be to write an informal explainer to document the ideas floating around sensor discovery. It could be just a simple markdown file somewhere e.g. on your own repo. If you'd like to make substantial contributions down the road, we'd be happy to have you join the group officially, see https://www.w3.org/2009/dap/#participate

[larsgk]

@anssiko , As this is something I've spent quite a bit of time on over the years, I'd love to help contribute to making it a solid solution - both for hobbyists and potential industry, who could benefit a lot.

Given that BLE Mesh implementations for home automation is moving now, that we have mobile devices interacting with a dynamic list of sensors and actuators and that we now have multiple ways of easily connecting external hardware to the browser(s), makes it a very interesting field indeed.

[anssiko]

@larsgk, happy to hear you're interested in contributing to this sensor discovery issue -- we're looking forward to your input, explainer or the like.

The wider W3C community will meet early November at TPAC and this topic would be a good fit to be discussed in the related breakout session, so I noted it explicitly in https://www.w3.org/wiki/TPAC/2017/SessionIdeas#Device_APIs_session

[larsgk]

@anssiko fyi, was a bit delayed by other 'events' but topic being worked on atm

[larsgk]

Some initial examples here: https://github.com/larsgk/imo/blob/master/GenericSensorDiscovery.md

I'll do some more concrete code examples + polyfills.

[Symbitic]

I hope this is still active. Has there been any progress on the basic interface for a future sensor discovery API? I'd like to take it into account when designing some libraries I'm working on.

Adding to the work of @larsgk, I have two new use cases and some possible solutions.

Use Case 6: Vendor-specific device subtypes

Real-world devices and sensors may not follow a simple browser -> attached/virtual sensor -> unique type flow. As an example: take the LEGO Bluetooth hubs. Each hub has ports that sensors/devices can be connected to, in addition to sensors integrated into the hub itself. A hub might have an accelerometer built in, and it might have a distance sensor plugged in on port A that exposes accelerometer and distance data. A vendor could have dozens of possible sensors with varying degrees of similar data.

Use Case 7: Multiple sensor sources

Building on use case 6, there may be more than one source of sensors connected at a given time. Obviously web browsers will never standardize a LEGO Bluetooth hub API, but if the goal is a standardized discovery API, then it should be extensible to cover userspace-specific instances.

Solution 1: Methods

This one was suggested above. Sensor discovery should be handled by a method that returns a async/non-async list of sensors:

navigator.requestSensors(...)

A solution like this would be simple and familiar to users of other Web APIs. The biggest problem would be how to handle a huge range of possible devices and use cases.

Solution 2: Events

Sensors could be globally announced using an event. The browser would handle built-in recognizable devices, while the user could dispatch the event manually for user-specific sensor types.

navigator.requestSensorScan(...);
navigator.addEventListener("sensorfound", sensor => {})

This solves the problem of multiple sensors coming online at random times, which will likely be important for data-intensive long-running applications. A method like requestSensorScan could be used to set parameters that would help with battery-saving operations while also prompting the user for permission to scan for devices.

Solution 3: Constructor parameters

This is easy enough. Just rely on additional parameters passed to the constructor of each sensor implementation.

new CurrentSensor(...)

Nice and simple, but doesn't fit will with asynchrony. There's also the issue of security.

Solution 4: Registry

With this solution, there would be a global registry of devices that could be populated by the browser and the user. Sensors could be a map similar to Web Components:

navigator.sensors.get("accelerometer-named-james-by-scitech");

Alternately, if a field like name and/or type was standardized as part of the generic sensor API, sensors could be a WeakSet instead:

for (const sensor of navigator.sensors) {
    if (sensor.name === "mydevice" && sensor.type === 1001) { ... }
}

Sensors would disappear from the registry once they are no longer available. This might also benefit from the recent work on custom registries for Web Components. On the other hand, it would almost certainly require events to announce that a new sensor is available.

[anssiko]

@Symbitic thanks for sharing these use cases and proposed solutions. As you probably noticed, there hasn't been progress on this issue lately. However, the fact that this issue is open means the group might resume this work. In the web standards land 10 years is not a long time, for better or worse.

As you implied, the group has indeed been focusing on browser as an implementation target thus more advanced features that do not cater to mainstream have not made it to the spec and implementations. The Generic Sensor API is extensible by design and allows, for example, instantiation of multiple sensors of the same type, or plugging in a discovery mechanism later. This differentiates it from legacy sensor APIs that came before it. We saw a need for these more advanced features, so wanted to offer a future-proof API. Certain browser vendors are still with the legacy sensor APIs, so it is beneficial to get everyone on board with the new baseline first before expanding the scope further.

That said, if you plan to do prototyping in the space of sensor discovery, we'd love to hear from you. Please feel free to drop pointers to this issue to your experiments. Thanks for your contributions!

[Symbitic]

@anssiko sure. I'll try experimenting with several approaches and report my experiments.

One thing I thought of: there is a certain amount of overlap between sensor discovery and WebHID. Since this is the Sensors and Device working group, maybe some kind of merge between Generic Sensor discovery and WebHID should be considered, either in the form of extending WebHID or creating a new HID that replaces WebHID and unifies devices and sensors.
For example, navigator.hid.getDevices() could be pre-populated with things like Magnetometer or Accelerometer after calling navigator.permissions.query. Generic Sensor is extensible and future-proof, but we shouldn't assume that the future holds that sensor and device are exclusive the way they are today.

[larsgk]

Hi @Symbitic - Thanks for sharing this!

One thing I thought of: there is a certain amount of overlap between sensor discovery and WebHID. Since this is the Sensors and Device working group, maybe some kind of merge between Generic Sensor discovery and WebHID should be considered, either in the form of extending WebHID or creating a new HID that replaces WebHID and unifies devices and sensors.
For example, navigator.hid.getDevices() could be pre-populated with things like Magnetometer or Accelerometer after calling navigator.permissions.query. Generic Sensor is extensible and future-proof, but we shouldn't assume that the future holds that sensor and device are exclusive the way they are today.

In general, discoverable sensors could have many possible types of connections - and they may go in and out of range, only be available in certain locations, with certain permissions, etc. and IMO should not be tied to the transport layer.

A first step might be to create the foundation for extensibility via e.g. drivers adhering to a specific interface to be brokers between external sensors (serial, hid, bluetooth, virtual, etc) and generic sensor class counterparts, available to applications.

The current generic sensor implementations are not suited for multiple sensors of the same type used in the same application. Registration, discovery, selection and usage should also be catered for in the APIs. This is something that should be possible to experiment with using polyfills today. Recently, I made an experiment to inject sensor data from an external device into a web application using the existing sensor APIs. Something that might be practical when developing applications using sensors not available on the device used for development - maybe some inspiration can be found there too? https://dev.to/denladeside/generic-sensors-and-thingy52-9oa

One other thing that might be interesting to look at in this area is the potential use of the Coordinated Set Identification Profile ( https://www.bluetooth.com/specifications/csip-1-0-1/ ) - try to read section 1, where sensors are mentioned.

I'd love to put some effort into reviving this topic and maybe you can give your thoughts on some of the things I've mentioned above?

[Symbitic]

@anssiko @larsgk I've done some preliminary experiments, and I put together an unfinished basic spec here: https://gist.github.com/Symbitic/7e7f0722da6357aaa2d5a2a2df999c4d

Long story short, it defines a generic SensorSource interface that acts as a driver, adds an optional source member to the Sensor constructor, and defines a SensorRegistry interface that handles creating instances of SensorSensor that can be passed to the Sensor constructor.

I'm still working on SensorRegistry, but the idea is that it will contain a list of categorized sensors while also allowing the user to register a way of detecting sensors (i.e. registerSensorScanner("Accelerometer", DiscoverBluetoothDevicesWithAccelerometers)). This should allow hosts (the browser) to securely expose multiple sensors while still being extensible enough to allow the user to add other sensors. Static (hardware) sensors can be pre-defined based on permissions, while dynamic (software) sensors can be added later.

No part of this is finished yet, but I'd like to hear your thoughts on the work so far.

[Symbitic]

I put more thought into this and significantly redesigned the proposed interface. Instead of working like low-level OS drivers, it is centered around sensor readings and a SensorBackend class to provide them. It's much more application-friendly as well as more secure, since permissions and data-handling are ultimately handled by the Sensor.

The idea came from Qt's sensor library. I've been using it for some recent projects and found it to be a natural and enjoyable way of interfacing with sensors. This should be ready for serious consideration.

https://gist.github.com/Symbitic/7e7f0722da6357aaa2d5a2a2df999c4d

Any thoughts?

cc @anssiko @larsgk

[anssiko]

@Symbitic thanks for your contributions. We'll discuss this topic at our F2F next week.

[Symbitic]

@Symbitic thanks for your contributions. We'll discuss this topic at our F2F next week.

Thank you. Please let me know if there's anything I can do to help.