[css-viewport-1] Bring the segments property into viewport

[darktears]

The segments property allows developers to target dual screens and foldable devices by giving information about the segments (or zones) when the browser window is spanned across the screen(s). There is a good set of pictures to demonstrate what we are trying to achieve here.

Work was started by Microsoft in WICG to bring a JavaScript API, link here. The proposed specification adds a segments property to the visualViewport object. segments are only defined when the browser window is maximized. They are also defined in CSS pixels. When the browser window state changes (maximized/resized) one needs to listen for the resize events to query the updated segments.

Please note that the segments property is intended mostly to mirror in JavaScript what’s being available in CSS (see MQs).

Both APIs, CSS and JavaScript went through TAG reviews here and here.

In terms of implementation status, Android and Windows are supported in Chromium based browser (with some caveats). Microsoft Edge does ship the CSS and the JavaScript APIs enabled by default on all platforms. Samsung Internet also is shipping both APIs by default on Android (the only OS they target).

[tabatkins]

This seems like a pretty reasonable addition to me. As far as I can tell it's just exposing directly in JS what is already exposed to CSS via the env() viewport segment variables, so I can't see any possible concern with this.

[tabatkins]

Is this issue asking for review of the segments property? Or of the VisualViewport class more generally (defined in https://wicg.github.io/visual-viewport/#dom-visualviewport-segments). Or is it asking for that spec to be folded into CSSOM? Just unclear what the request for this WG is here.

[darktears]

This seems like a pretty reasonable addition to me. As far as I can tell it's just exposing directly in JS what is already exposed to CSS via the env() viewport segment variables, so I can't see any possible concern with this.

You're right, that's exactly this.

[darktears]

Is this issue asking for review of the segments property? Or of the VisualViewport class more generally (defined in https://wicg.github.io/visual-viewport/#dom-visualviewport-segments). Or is it asking for that spec to be folded into CSSOM? Just unclear what the request for this WG is here.

VisualViewport is already in the CSSOM spec draft (see here). I'm asking to add the segments property in VisualViewport which isn't there yet.

[tabatkins]

Ahhh, I was looking in CSSOM, not CSSOM View (that's a separate spec). Okay, let me just fix the title/labels, I understand now.

[darktears]

Ahhh, I was looking in CSSOM, not CSSOM View (that's a separate spec). Okay, let me just fix the title/labels, I understand now.

My bad, I didn't read carefully the name of the spec.

I'll wait more folks to comment before I put a PR together, especially the editors of the spec.

[bokand]

FYI: I moved the visual viewport spec into CSSOM-View but didn't include segments since I wasn't sure how widely accepted that was and it was still marked as "experimental". I'm guessing you'd want some support from another engine?

[darktears]

FYI: I moved the visual viewport spec into CSSOM-View but didn't include segments since I wasn't sure how widely accepted that was and it was still marked as "experimental". I'm guessing you'd want some support from another engine?

Yes ultimately we want to move this property further along in the standardization process.

[darktears]

@fantasai said

Adding new features (or substantially changing existing ones) requires CSSWG approval, not just editor approval... it doesn't seem like #9237 has that?

How do I request CSSWG approval on the added property?

[darktears]

Summary of the discussions on the PR (for discussions in the conf call):

• Fingerprinting: what does this adds more than what currently can be accessed/deduced.
• Exposure to iframes: The CSS MQs are not restricted now, and they expose the same values. So far, I couldn't find an example where MQs are restricted to iframes. Why this one should be? Developers can still use JavaScript to access the CSS MQ and env() values.
• Should this be part of the VisualViewport API? I think it should, otherwise where?
• Explain more in detail what exactly the FrozenArray of DOMRects returns especially regarding zooming and panning.
• Where to put the automation for WPT? We need to emulate display features (fold/physical hinges) so we can check the segments are correctly calculated. This would be something like https://www.w3.org/TR/device-posture/#automation. This automation section would also be useful to validate the CSS API so maybe there is a better place for it.

[css-meeting-bot]

The CSS Working Group just discussed [cssom-view] Bring the segments property into visual viewport , and agreed to the following:

• RESOLVED: define the segments on a new layout viewport object on window tbd

The full IRC log of that discussion

<jarhar> what is alexis's username? if i dont have anything ill just say "alexis"
<emeyer> scribenick: jarhar
<jarhar> alexis: one of the gaps we have is support for layout in pop? and making content is not on the full section, and there was work done in the past by ms for css ??? and its already part of the media query
<jarhar> alexis: the bottom section of the screen, bringing parity to support the functionality in the javascript world
<jarhar> alexis: lets say youre in a canvas or a game, you cant necessarily use css so in the past we used ??? and css and javascript, that was the PR in the bug
<astearns> zakim, open queue
<Zakim> ok, astearns, the speaker queue is open
<jarhar> alexis: requesting approval to add this feature to the ??? spec, and there were a couple questions raised
<jarhar> alexis: i went to great lengths to show how were not exposing anything new
<astearns> s/???/CSSOM view?
<jarhar> alexis: there was a question about iframes and media queries
<jarhar> alexis: there was a question about whether this should be part of ???, and WPT support and webdriver support for us to test this
<jarhar> alexis: we need webidl support
<jarhar> alexis: thats basically it, im here for questions
<TabAtkins> +1 to this in general (i've been bad about supporting this issue)
<jarhar> alexis: the css viewport segment as well as the viewport - as well as the second compnent of visual viewport are in origin trial, we are seeing feedback on this feature, went through tag reviews
<astearns> q+
<miriam> ack astearns
<jarhar> astearns: i wonder whether we could resolve on adding this to the cssom view spec, and create separate issues for each of the things that you got in your summary
<jarhar> astearns: we dont have to figure out if it has to be aprt of the visual viewport api, but if its in a draft you might get more traction on individual issues
<jarhar> alexis: yes, i would like to resolve this at some point
<bramus> q+
<miriam> ack bramus
<jarhar> bramus: i havent read into what is in the spec, but im confused right now. do the segments expose the various viewports or the visual viewport?
<jarhar> alexis: they are multicol subdivisions of the visual viewport
<jarhar> alexis: for fullscreen, its logical subdivision of the visual viewport. im not sure if fingerprinting was an issue
<jarhar> alexis: that includes also figuring out the resolution of laptop
<jarhar> bramus: what about window.visualViewport? it doesn't get affected by page zoom
<jarhar> alexis: the only thing that would take into account is the scale factor of the device, thats the only thing we do
<jarhar> bramus: there is an issue 8709 where it was mentioned to have somtehing like window.viewport eventually, and maybe you should work on that where we expose the viewport. you have to postfix somewhere, we could tackle both in one go
<jarhar> bramus: and then you have segments as you described here
<jarhar> alexis: the reason it was there is that i wasn't the original author editing there, but because the visual viewport ?? are basically equivalent for intersection
<jarhar> alexis: so thats why it was built that way, it was visually speaking you see two segments, thats probably why it was put there in the first place
<jarhar> alexis: i agree with you, its not a window property
<jarhar> miriam: can we take the resolution alan suggested and then open issues?
<jarhar> bramus: we might want to move this somewhere
<jarhar> astearns: we would only be stuck if people started implementing and we got web content that relies on it. we need to have some draft text to be working on it
<flackr> q+
<jarhar> astearns: this will help us get more attention
<jarhar> alexis: if we move it then i will have to redo the origin trial
<jarhar> alexis: ??? css api as well as the js api, implementing a full fledged support on your app,
<jarhar> alexis: at some point we will ship the rest of the pieces
<jarhar> alexis: implementation in chromium, origin trial just for that js api, we are already late in terms of spec
<jarhar> alexis: speaking as an implementor, not as the spec side of things
<miriam> ack flackr
<jarhar> flackr: this isnt an objection to this proposal, but that got me thinking about what pinch zoom means
<jarhar> flackr: im wondering if we need some better way to have multiple viewports
<jarhar> flackr: would you want to pinch zoom a segment of the display?
<jarhar> flackr: you probably dont want it to move content on one segment to another segment
<jarhar> alexis: true, but pinch to zoom implementation is stuck to the page
<jarhar> alexis: i see what you mean but then we are going to an implementation complexity thats crazy
<jarhar> alexis: do i want to zoom the app? or one section only?
<jarhar> flackr: im just trying to figure out if we could design it any way we wanted then where would we end up?
<jarhar> alexis: then how would we do this in css? theres the css part we have ??? so how would you...
<jarhar> flackr: this isnt an objection to your proposal, i think its fine
<jarhar> flackr: i was just throwing out the idea that maybe we should think of a css way or js to define viewports within your viewports or something like that
<jarhar> flackr: im not objecting to exposing what we have
<jarhar> alexis: but my question about pinch to zoom: what happens if you pinch to zoom? ok then dont change
<jarhar> flackr: you would be zooming into a particular viewport not the root visual viewport
<jarhar> alexis: the concern with the visual viewport - what happens with escape popover?
<jarhar> flackr: the device scale factor doesn't change
<jarhar> alexis: so we would not change anything, we would not expose the viewport
<jarhar> alexis: its not aligned for contet, but its the same for any page thats out of viewport
<jarhar> flackr: the visual viewport api is exposing the visual viewport, but the thing youre trying to expose is outside of that, the full browser viewport
<jarhar> alexis: one possible implementaion would be for me to resnap these values to one is visible and one is pinch to zoom right?
<jarhar> flackr: i dont know how that would work i would need to see an example
<jarhar> alexis: with pinch to zoom any portion is what i can see after the pinch to zoom
<jarhar> flackr: yes
<jarhar> alexis: lets see i pinch to zoom, ok i understand what you mean yes
<jarhar> miriam: are we ready to take a resolution or do we need to take this back to the issue?
<jarhar> bramus: what i would like to see and from peopple at apple would be: if you did window.viewport or window.??? would we add some more things to that
<bramus> s/window.???/window.visualViewport
<jarhar> flackr: right now windows kind of our container for this kind of thing
<jarhar> flackr: would it make sense to have more things on window?
<emilio> window.viewportSegments?
<jarhar> bramus: maybe this would be our opportunity to correct some things from the past and ???
<bramus> s/and ???/and group it under window.viewport
<jarhar> fantasai: i dont know much about this model
<jarhar> fantasai: the one thought in myu head was about what coordinate system these are in, whether its zoomed coordinates or not and how does that related to other objects thats attached to
<fantasai> s/this model/the CSSOM/
<jarhar> alexis: right now its ??? its just the device pixel value
<jarhar> bramus: you have the viewport, you draw a line in the middle, and the visual viewport still bounces around, you can show parts of various segments because its a subsection of the entire viewport
<jarhar> fantasai: wouldnt you and an author want segments of the entire viewport not the visual viewport?
<jarhar> flackr: yes
<jarhar> bramus: i think thats what we currently have
<jarhar> emilio: but in css pixels not device pixels?
<miriam> q?
<jarhar> fantasai: where is that defined?
<jarhar> alexis: right now the visualViewport property
<emilio> fantasai, https://github.com//pull/9285#discussion_r1376720482
<jarhar> alexis: proposal to add a property in the visualViewport object
<jarhar> fantasai: where are we representing the segments in terms of the overall viewport?
<jarhar> fantasai: which object in js right now tells me that the hinge or whatever is 40% through the viewport?
<jarhar> flackr: my understanding is that its just css properties
<jarhar> alexis: you have two media queries that tells you the orientation and then you have css hover and variables that sets you each of the segments, including the width height in x and y
<jarhar> alexis: for each of the segments
<jarhar> fantasai: and those are required to the layout viewport
<fantasai> https://drafts.csswg.org/css-env-1/#viewport-segments
<jarhar> fantasai: so what youre wanting to add is a method to access this information with rspect to eh visual viewport
<jarhar> alexis: yes and being able to use this in js without css in the first place like canvas
<jarhar> flackr: its not with respect to ??? its respect to the viewport
<jarhar> fantasai: whats the use case with knowing that this is with respect to teh visual viewport since we are doing the layout viewport
<jarhar> flackr: there has been confusion, this doesnt belong in the ?? api
<jarhar> fantasai: then it doesn't belong on the visualViewport object
<jarhar> bramus: that was my point
<jarhar> alexis: but where then
<astearns> s/???/visual viewport/
<jarhar> bramus: id say window.viewport
<astearns> s/??/visual viewport/
<jarhar> bramus: and a way in javascript to get the small viewport height and the ??? viewport height
<jarhar> fantasai: does window.viewport exist already? and you want to create this that represents the visual viewport? that seems reasonable
<jarhar> fantasai: then i think the way forward here is to sketch out what this object would look like and include the segments information as well as the other information that bramus is mentioning
<emilio> This would probably be a good option to fix https://github.com//issues/5260
<jarhar> alexis: if you need help to sketch out then whats the work ??? i would need help on writing the dom
<astearns> did I hear bramus volunteer to do that?
<jarhar> miriam: we could take a resolution that this is the path forward
<jarhar> bramus: i would take a stab at that
<jarhar> alexis: we can connect offline bramus if you want
<jarhar> fantasai: proposed resolution: define the segments on a new layout viewport object on window tbd
<jarhar> miriam: any objections to that proposed resolution?
<jarhar> RESOLVED: define the segments on a new layout viewport object on window tbd