Screen Enumeration API

[bunneyster]

こんにちはTAG!

I'm requesting a TAG review of:

• Name: Screen Enumeration API
• Specification URL: TBD
• Explainer (containing user needs and example code)¹: https://github.com/spark008/screen-enumeration/blob/master/EXPLAINER.md
• GitHub issues (if you prefer feedback filed there): https://github.com/spark008/screen-enumeration/issues
• Tests: https://github.com/web-platform-tests/wpt/tree/master/screen_enumeration
• Primary contacts (and their relationship to the specification): @sPark008 (original API designer), @msw (implementer)

Further details:

• Relevant time constraints or deadlines: None
• I have read and filled out the Self-Review Questionnare on Security and Privacy. The assessment is here.
• I have reviewed the TAG's API Design Principles
• The group where the work on this specification is: TBD

We recommend the explainer to be in Markdown. On top of the usual information expected in the explainer, it is strongly recommended to add:

• Links to major pieces of multi-stakeholder review or discussion of this specification:
• Links to major unresolved issues or opposition with this specification:

You should also know that...

This API is intended to be used in conjunction with the Window Placement API, which is still in the proposal stage.

We'd prefer the TAG provide feedback as (please select one):

• open issues in our GitHub repo for each point of feedback
• open a single issue in our GitHub repo for the entire review
• leave review feedback as a comment in this issue and @-notify [github usernames]

---

Please preview the issue and check that the links work before submitting. In particular, if anything links to a URL which requires authentication (e.g. Google document), please make sure anyone with the link can access the document.

¹ For background, see our explanation of how to write a good explainer.

[torgo]

Discussing in the f2f right now - @lknik can you take a look at the answers to the privacy & security questionnaire?

[hober]

Labeling as a Fugu-related request since it appears in Fugu's full list of capabilities.

[lknik]

Each of the new display properties chosen for this API makes a non-replaceable contribution. Removal or constraining of any property may degrade the user experience. Some properties offer more value than others, and some are more relevant to the proposed use cases than others. Care should be taken to choose those properties offering the most value to the most important use cases.

It has been suggested that user agents themselves could provide a UI for selecting where content is presented, but this is cumbersome for users and prohibitively limiting for web applications. This also offers no meaningful value over the existing cumbersome manual placement of windows by users.

Can't we simplify to "for each display we need same information as is currently offered by existing screen object"? We do not discuss the contents of the screen object, but the inflation of (many) screen objects.

2.3 This API does not expose such information. Any exposed information may identify the machine, but not its user.

Information allowing to identify the user's machine is more or less identifying the user, so you're saying the opposite. Maybe at least remove the 2nd sentence?

2.6 This API proposes exposing some additional properties for each of the displays connected to the device; see Section 2.1 for a complete list.

The complete list seems to be X*12-15 static objects (some of them dynamic; with X the number of screens).

2.7 Does this specification allow an origin access to sensors on a user’s device? No.

It does on the other hand inform if the display has an accelerometer, which would make it simpler to direct a malicious script to an appropriate window.

2.8 "The following display properties are currently exposed ..."

So we can today access all the device screens already? Wouldn't it be more useful to enumerate the new identifiers to be exposed, which is the above 12-15 numbers per each screen, which is not currently easily available?

To mitigate this issue, user permission is required to access the display

Do I see it correctly that this API will be gated behind permissions?

[hober]

Marking as pending external feedback; @lknik‘s comments above have not been addressed, and we’ve also raised webscreens/screen-enumeration#23.

[plinss]

Looked again during Wellington F2F, still waiting for external feedback. We also have concerns about how this intersects with ongoing work for foldable screens #472.

An additional observation is that there's an explicit non-goal of exposing all the OS screen APIs yet this proposal seems to do so.

[hober]

@sPark008 I'm not sure if you saw @lknik's questions above. Could you take a look at them and get back to us?

[michaelwasserman]

Thank you for poking this thread and apologies for the delay!
I'll try to address the open questions and feedback this week.

[michaelwasserman]

Thank you for your feedback and patience! I hope you find my responses and changes helpful, and I look forward to iterating further with additional input.

@hober thanks for your feedback and patience, I hope you find these responses helpful:

We also have concerns about how this intersects with ongoing work for foldable screens #472.

This work is complimentary to Foldable Screens #472. That proposal offers simple tools to support content spanning the single fold of a dual-screen device. Screen Enumeration and Window Placement (TAG review request imminent) aims to support a broader range of multi-screen and multi-window scenarios with lower-level information (eg. 3+ screens, 2 heterogeneous or misaligned screens, separate windows on separate screens, etc.).

An additional observation is that there's an explicit non-goal of exposing all the OS screen APIs yet this proposal seems to do so.

The goal is to expose a reasonable set of info for web applications to make informed use of the available screen space; that will require exposing some limited new information, but care has been taken to select the most valuable subset of available information. The spirit of the rephrased non-goal is to avoid exposing extraneous or especially identifying information (eg. EDID), and avoid exposing APIs to control the configuration of display devices.

we’ve also raised webscreens/screen-enumeration#23.

I posted a couple comments there, and want to explore ways to support requests for limited screen information. Please let me know if you have any thoughts regarding my comments there.

@lknik Thank you for your Security and Privacy review and comments, I hope you find the latest changes there and these responses helpful:

Each of the new display properties chosen for this API makes a non-replaceable contribution. Removal or constraining of any property may degrade the user experience. Some properties offer more value than others, and some are more relevant to the proposed use cases than others. Care should be taken to choose those properties offering the most value to the most important use cases.

It has been suggested that user agents themselves could provide a UI for selecting where content is presented, but this is cumbersome for users and prohibitively limiting for web applications. This also offers no meaningful value over the existing cumbersome manual placement of windows by users.

Can't we simplify to "for each display we need same information as is currently offered by existing screen object"? We do not discuss the contents of the screen object, but the inflation of (many) screen objects.

I modified the section to separate the goals of exposing multiple screens and new screen properties , both of which are valuable.

2.3 This API does not expose such information. Any exposed information may identify the machine, but not its user.

Information allowing to identify the user's machine is more or less identifying the user, so you're saying the opposite. Maybe at least remove the 2nd sentence?

You're right, I revised the section to accurately convey that new information is exposed that could be used to help identify a device, and mention the possibility to gate access upon user permission.

2.6 This API proposes exposing some additional properties for each of the displays connected to the device; see Section 2.1 for a complete list.

The complete list seems to be X*12-15 static objects (some of them dynamic; with X the number of screens).

I added a comparable summary in this section and added screen.[avail]top/left in section 2.1.

2.7 Does this specification allow an origin access to sensors on a user’s device? No.

It does on the other hand inform if the display has an accelerometer, which would make it simpler to direct a malicious script to an appropriate window.

I added a note to this effect in the section.

2.8 "The following display properties are currently exposed ..."

So we can today access all the device screens already? Wouldn't it be more useful to enumerate the new identifiers to be exposed, which is the above 12-15 numbers per each screen, which is not currently easily available?

I rewrote this section to give a more pointed answer to its question, including the difference of the single screen available to each window and the full set of screens, and a summary of what's currently available for each considered/proposed property.

To mitigate this issue, user permission is required to access the display

Do I see it correctly that this API will be gated behind permissions?

Yes, user permission should be a critical consideration of any browser that might adopt something like this proposal, and the asynchronous getScreens() should enable user-facing permission prompts.

[michaelwasserman]

Please consider #522 alongside this proposal, they're highly related; thank you!

[michaelwasserman]

You may wish to consider these specific questions during review. Thank you!

• Reuse existing Screen interface or add something new? webscreens/screen-enumeration#31
• API design for a single multi-screen boolean API design for a single multi-screen boolean webscreens/screen-enumeration#30
• Combine Screen Enumeration and Window Placement proposals? Combine Screen Enumeration and Window Placement proposals? webscreens/screen-enumeration#34
• Reject the getScreens() Promise with permission denial Reject the getScreens() Promise with permission denial webscreens/screen-enumeration#29
• Scope of the initial proposal Scope of the initial proposal w3c/window-management#15

[hober]

We talked about this during a breakout today. We'll leave comments on each of these issues.

[michaelwasserman]

I believe I have addressed all recent feedback, thank you!

[kenchris]

Hi there,

We are fine with the work you have done and we are therefore going to close this issue. @plinss is still a bit worried that part of this might ship in a way that it is incompatible with the work around foldables, and will file an issue about that on your new repo.

Thanks for staying at home with the TAG!