[css-media-queries] Foldables support and enablement

[argyleink]

Why

Android Q beta has support for folding the screen in the emulator and a list of features and APIs to support developers in using and adapting to foldable screens, I'd like to see the web have similar offerings in the form of a new @media query or queries.

This new device/OS state is not the same as orientation or device width. It's more than that, as a webapp could project content while in a folded state onto the back or side of a device.

Imagine an <aside> literally on a foldable side of a device while reading an article.

What

As I dig into the Android featureset and APIs, the nature of the media query feels akin to how devs work with projectors and orientation.

Designers and developers should be able to leverage or adapt to these foldable device states and either project content onto another half of the device, or just simply adjust to the wider or thinner viewports respectively. There are many options, designs, patterns, etc for how to handle and adjust to foldable phones, all of which are being pioneered or explored right now as the technology unfolds (😉).

From the Android blog:

On this new form factor, your application could be transitioned from one screen to another automatically (eg. when folding / unfolding a foldable phone).

During this transition, your app will receive a configuration change for the new layout (and possibly density in some cases).

The ability to handle this transition shouldn't be limited to native apps, the web should have hooks into handling this as well.

How

These are just my initial aggregated thoughts and assumptions from my research on which media queries would be useful. I hope to expand upon it greatly to cover the matrix of possibilities and unblock engineers on the web.

1. is foldable screen
2. is consuming all screens
3. is folded vertically
4. is folded horizontally
5. is on main screen
6. is front / is back
7. is on screen (1-n)
8. can project onto screen (1-n)

I envision these media queries assisting and enhancing:

• css env vars
• regions!
• device width and height media queries
• portrait and landscape media queries

Conclusion

I'm feeling like this media query could be very complex, as it needs to provide syntax for a wide array of foldables. That being said, the types of experiences and adaptations that web pages could do with the display hooks would make for very interesting and meaningful interactions. We should work towards enabling the web to handle the upcoming array of devices.

Links/References

• https://www.w3.org/TR/css3-mediaqueries/
• https://developer.android.com/preview/features/foldables

[tabatkins]

Yeah, those diagrams showing a 3-panel folder make me very nervous about designing a MQ here. Gives me flashbacks to trying to generalize about a rounded-screen MQ, which proved impossible.

The core question we have to answer here is, what questions should we enable the author to ask, such that they can do something useful with the answer? I'm not seeing very many sorts of questions I could ask that would let me usefully design to both a side-folding phone and a 3-pane tablet. :(

A JS API can be more fine-grained in answers, perhaps.

There might still be some stuff we can usefully extract here, tho. This could use some exploration. Like, maybe we can catch just "the X edge is folded", with a fallback "lol this is complicated good luck" value.

[tabatkins]

Oh, and the "main screen" vs "secondary screen" dichotomy seems potentially useful. I'd want to see a worked-out example that would convincingly work on different devices using that first, tho.

[AmeliaBR]

I agree that this needs to go beyond media queries. You need to be able to target the different parts of the screen for layout in some way. At the very least, there would need to be environment variables defining the areas and their dimensions, so that you could create a CSS grid on your root element that defines the different screen panels as different grid areas.

And it makes sense to approach this in a more general way, and consider the ability to have a single website control two screens for working with dual monitors or a projector. Another device with a secondary screen is the mac "touch bar" screen. I think apps have a way to draw content here, I'm not sure if there is any interest in exposing that screen to websites (or if it is better to let it be used for browser controls).

But, we need to distinguish between second screens/panels that should only be used for sidebar/controls, versus those where it makes sense for the main body context to fragment across & fill up the extra screen. With a folding screen, this could be the difference between a bent screen (like a book, you might want a column of text on each side, paginated to fit the screen) and one completely folded over (you could put the nav on the back, but might want to have the main text on the front screen in a scroller).

It's also worth thinking about how much user-control there should be. I assume there would be a user setting of the form "if my screen is folded, only use the front screen" vs "use both screens". Are there other relevant user preferences? If so, how should these variations be exposed?

Just from this discussion, I think there are two incomplete areas of CSS layout that need to be resolved to make these layouts really useful: CSS regions (fragmented text across grid regions representing panels of different sizes), and for screens.

[fbartho]

Is it really crucial that this is a "Foldable" concept? Or is it rather more general to talk about a series of viewports that you might be able to display into, and the geometry of those edges?

Aka if I have 2 panels, can I choose to display a seamless thing across them? or are they distinct screens that aren't close to each other?

[tabatkins]

There is a foldable-specific aspect to this; displaying on the folded screen edge is different from displaying anywhere else.

But yes, other than that this is a more generic "i have multiple display surfaces" problem.

[fbartho]

Right, so I meant to elaborate my idea, you could have any number of viewports, that would attach to each others in different ways.

Additionally, it's not hard to imagine a multi-display device, that slides one screen behind another (locking at different positions) in some circumstances, rather than "folding". -- As long as the metaphor covers different edges cases like this, then I don't exactly "care" if it uses the word foldable in the name, I just wanted to make sure we didn't design this without thinking about things that aren't strictly folded into place.

[patrickhlauke]

If the different folds are meant to work independently, it feels more like you'd definitely want something more akin to separate browser instances/windows that can still cross-communicate, rather than a single page. To me anyway...

[AmeliaBR]

For the specific issue of avoiding putting content on the fold itself (the seam/spine as the screen wraps around a bend), this seems to be similar to the "safe inset" regions for notched iPhones. Within each screen panel, there will be some parts of the screen that you could paint a background on, but wouldn't want to put important content.

However, the finite set of safe inset margins currently defined for CSS environment variables falls apart if you need margins for many different layout regions.

[tabatkins]

Yes, I assume that, in cases like the folded-phone where it's just "small screen with folded edge" vs "large screen, normal edges", we can specify that safe-inset will communicate the size of the fold-region so you can avoid it if that's all you're worried about.

---

So my thoughts right now are:

1. Specify that env(safe-area-inset-*) should take folds into account, letting you avoid positioning anything on the fold.
2. Might be useful to add a fold-side: [top | bottom | left | right | inline-start | ...] MQ, which can match multiple of those values, to communicate which edges are folded if you want to adapt your UI to take advantage of the fold.
3. Investigate the multiple-display-area issue: how are they exposed to pages on current devices, what information might authors need from that, and is that information amenable to MQs or more suited for a specialized JS api.

[argyleink]

should we open a separate issue for safe-area-inset types for foldables? or let it stay within this context of media queries?

[AmeliaBR]

Saw this tweet today announcing a Microsoft surface phone/tablet with split screen. Not technically a "foldable", because it uses two distinct screens, but it looks (from the limited info available) like the intent is for some content to spread across the two "pages", which raises the same potential problem of having content on the split where it isn't useful.

They're target shipping date is November 2020. It'd be great if some of our Microsoft folks could make sure web standards are looped in on any proposals for exposing the screen layout to applications. Since the device will apparently be Android based, it would be ideal if the solutions also worked for other foldable Android devices like the ones presented earlier in this thread.

[AmeliaBR]

From the Microsoft team, looks like they do have a proposal for a DOM API, so that might be a first place to start discussions. If many teams agree that this covers all the information needed for different device types, then we (CSS WG) can start talking seriously about how that would look in a declarative context: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/Foldables/explainer.md

[zouhir]

I am one of the authors of the explainer linked above and we have just published a bunch of updates, the main one is around adding CSS primitives to enable developers detect and react to screen spanning declaratively in CSS.

In a brief summary, the updated explainer has 2 new CSS constructs:

1. spanning media query
   - allows developers to test whether their content is spanning across 2 screens.
   - informs them whether the screens are "stacked" or "aside" using the device hinge posture.
2. a set of browser-defined env() variables
   - to inform developers about the size and the position of the hinge so they can calculate the size of each screen segment.
   - since foldable devices can be running different OSs (Android, Windows, etc), and the underlying OS might (or might not) choose to draw pixels behind the physical gap, we chose to represent the hinge as a 2d rect, the width or height of that rect can be 0px if the underlying OS "scissor cuts" the content and does not draw behind the hinge.

Here's the link to our updated explainer (same link as in the comment above), would love to discuss it with you folks and hear your feedback: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/Foldables/explainer.md

[argyleink]

latest of @zouhir's work https://github.com/zouhir/spanning-css-polyfill

[zouhir]

I've opened a new issue #4736 with some thoughts behind what are we proposing and why; we can also use it for discussions and feedback that are direct to our proposed primitives.