Controlling screen brightness

[willmorgan]

Hello WebKittens (@marcoscaceres)!

Your input is welcome on the ability to control screen brightness, and if it is positive, then further input on how we should expose that on the web platform would be valuable.

• Spec Title: Controlling screen brightness
• Explainer (if not README.md in the repository): https://github.com/w3c/screen-wake-lock/blob/gh-pages/brightness-mode-explainer.md

Feedback and positions from the community and other browser vendors is being collated below:

w3c/screen-wake-lock#335

[hober]

cc @rniwa @othermaciej @jernoble @smfr @litherum

[rniwa]

Two use cases listed in the explainer sound rather bogus:

a user scans a QR/barcode under various lighting conditions, in which case boosting the screen brightness creates more contrast.
increasing the screen brightness can help illuminate a user's facial features, improving the viability of certain application that rely on the front facing camera

Given that the user can already manually adjust screen brightness via control center on iOS, I really don't see why we need to provide a specific API for this. Increasing brightness of the screen to improve the contrast of user's face sounds specially questionable proposition.

[willmorgan]

Two use cases listed in the explainer sound rather bogus

I would kindly point out that those particular use cases are in production today in the solution my team has built, and deliver remote onboarding to a few million users a week 😉

Given that the user can already manually adjust screen brightness via control center on iOS, I really don't see why we need to provide a specific API for this. Increasing brightness of the screen to improve the contrast of user's face sounds specially questionable proposition.

The idea behind programmatic control is that we can improve UX by doing this, particularly in full screen, for a temporary and short period of time.

For WebKit specifically this would improve completion rates due to the fact that the alternative way to improve signal strength is to go into full screen, which isn't possible with WebKit on iOS unless the element is a video.

[beaufortfrancois]

Two use cases listed in the explainer sound rather bogus:

• a user scans a QR/barcode under various lighting conditions, in which case boosting the screen brightness creates more contrast.

This feature is used by COVID apps to automatically go to maximum brightness while presenting a QR Code in fullscreen mode. The "TousAntiCovid" french gouvernement app is one example:

• https://gitlab.inria.fr/stopcovid19/stopcovid-android/-/blob/master/stopcovid/src/main/assets/Strings/strings-en.json#L1078 for Android
• https://gitlab.inria.fr/stopcovid19/stopcovid-ios/-/blob/master/TousAntiCovid/Resources/Jsons/Strings/strings-en.json#L1078 for iOS.

• increasing the screen brightness can help illuminate a user's facial features, improving the viability of certain application that rely on the front facing camera

Makeup Mirror apps are actually pretty popular: https://play.google.com/store/search?q=Makeup%20Mirror

[tomayac]

Another example are ticketing apps, for example, DB Navigator by Deutsche Bahn.

[smfr]

Maybe limiting this to content that's in fullscreen would be OK.

[willmorgan]

Maybe limiting this to content that's in fullscreen would be OK.

Would you be in favour of this being added as an option inside the requestFullscreen API?

If so, do you think it's worth reconsidering the current video-only limitation on iOS fullscreen elements?

Also, we would need to look at how to tell if the brightness request succeeded or failed through that API. The WG hasn't looked at that in much detail yet.

[marcoscaceres]

@willmorgan, can you elaborate on:

we would need to look at how to tell if the brightness request succeeded or failed through that API.

What did you have in mind there?

[willmorgan]

@marcoscaceres This proposed functionality, probably on the sentinel, would tell the developer whether or not the request succeeded (but not necessarily why).

From @jwheare at w3c/screen-wake-lock#335 (comment)

a simple boolean for isMaxBrightness would be helpful to avoid unnecessary user prompts while mitigating high resolution fingerprinting attempts

[marcoscaceres]

Ah! thanks for clarifying, @willmorgan! I think that ultimately depends on what shape or form the solution takes based on the choices in the explainer (i.e., if the brightness control is decoupled from the screen wake lock or not).

[marcoscaceres]

I'm personally opposed to this proposal (i.e., this position is NOT representative of the WebKit community; I'm posting it as input while waiting for further input from WebKit community members).

I'm concerned that allowing arbitrary adjustments to brightness, even if limited to a single "max" value, could be open to abuse - even if gated by user activation. Further, gating on .requestFullscreen() would require sites to change their designs, which I feel would be undesirable for developers and burdensome for users. Given the limited number of use cases presented, I don't feel this warrants a new API or changes to existing APIs.

For the use cases that center around QR and barcodes: the user agent could automatically detect when a QR/barcode is on screen and take appropriate action. This is not a hypothetical: for example, Safari is already capable of identifying QR codes within web content.

For the use case relating to biometric authentication based on light from the screen: I'm of the opinion that Web Authentication API, and the purpose-build hardware on which web authentication relies, is better suited for this task.

[tomayac]

For the use case relating to biometric authentication based on light from the screen: I'm of the opinion that Web Authentication API, and the purpose-build hardware on which web authentication relies, is better suited for this task.

Just to clarify, this is an important use case for banks and the likes who need strong guarantees they are dealing with real human beings who actually are who they say they are.

Typically during the sign-up flow, they make you hold your passport into the camera, read a random sentence, twist your head, etc. Commonly, this flow requires downloading apps like POSTIDENT (common for Germany).

If the user could stay on the Web for the entire flow from sign-up to verification, this would be a lot better and increase the success rate.

[beaufortfrancois]

Thanks for your input @marcoscaceres!

I'm are concerned that allowing arbitrary adjustments to brightness, even if limited to a single "max" value, could be open to abuse - even if gated by user activation.

Which abuse do you foresee when limited to a single "max" value when gated by user activation?

[willmorgan]

@marcoscaceres Thanks for clarifying your position. As others are focusing on the actual brightness API and abuse concerns, I'm just going to deal with why the iProov use case complements rather than competes with the Web Authentication API and FIDO technology in general.

• First off, FIDO is great at quickly, cheaply authenticating a specific device. However, registering ("binding") a device needs to be done securely.
• Secure binding is more, not less important with the advent of multi device credentials. You need to assert that the human using the device at the point of FIDO registration is who they claim they are.
• In some cases, FIDO biometric factors may fall back to PIN, and certain 1-N biometrics like fingerprints can't guarantee the person is the same.
• As @tomayac mentions, FIDO is great for a single aspect of authentication, but it doesn't currently deal with matching a biometric identity to government identity documents, etc.
• Organisations may require proof of identity at the point of authentication for regulatory and compliance requirements. A successfully completed FIDO ceremony, on its own, doesn't fulfil those requirements.

As for "purpose built hardware": our web based use case intends to use platform features like HSMs, secure enclaves to sign and store keys that can be used to tie the Web Authentication API to the wider, biometric information.

Our iOS SDK (in Swift) currently utilises all of these capabilities - screen brightness control, HSM, etc.

I would be glad to go through in a bit more detail any more questions you have!

[voxpelli]

As the following demo shows, it's already possibly to play around with max brightness in Safari by abusing the HDR-capabilities of the browser: https://kidi.ng/wanna-see-a-whiter-white/

Depending on where the HDR-support on the web is heading and which hacks or non-hacks it enables over time, an API like this could be seen as a replacement for a hack rather than an introduction of a new capability into the platform.

Just wanted to chime in with this added perspective.

[marcoscaceres]

@beaufortfrancois wrote:

Which abuse do you foresee when limited to a single "max" value when gated by user activation?

Just boosting brightness unnecessarily or going against the a user's preferences. For example, a site might decide that, "you know what? our site looks super nice when it's on full brightness... we should just ask the user for permission to boost brightness when they view our content because OMG 🤩 so pretty!".

Definitely don't want to end up in the above situation.

@tomayac, wrote:

Typically during the sign-up flow, they make you hold your passport into the camera, read a random sentence, twist your head, etc. Commonly, this flow requires downloading apps like POSTIDENT (common for Germany).

I don't disagree and I've had to do this myself a number of times here in Australia too. However, boosting the screen brightness is still a limited substitute to achieve the desired outcome of authenticating a person in the manner you describe. I think we can do better if, for instance, (just spit-balling here) one could tell the browser what the intent of the interaction is and the user agent can do the right/best thing there.

For example, the developer could hint to the browser that it wants to take a picture of a person. Then, the browser could bring up the front-facing camera and instruct the user to hold the phone at the right distance away. The user agent could then adjust the brightness by using both the camera's aperture and the screen as a flash, based on lighting conditions if needed.

There are already applications that can detect if the person is in frame for the purpose of authentication:

And then knows when your face is in the right place:

One could maybe do something like the above also with video embedded in a page?

If the user could stay on the Web for the entire flow from sign-up to verification, this would be a lot better and increase the success rate.

Yeah, exactly. We have the same goal. I just don't think that by just boosting brightness is how we get there tho.

@willmorgan wrote:

FIDO...

I'm in agreement about all the things you mention. I just think boosting brightness is only solving part of the problem. I'm not saying we need to do everything in one API. Just that it feels like this doesn't address the use case in a very elegant/webby manner (when compared to things like Payment Request, Web Authentication, and other APIs).

If this is an important use case, I'm asking if we could do better?

@voxpelli wrote:

Depending on where the HDR-support on the web is heading and which hacks or non-hacks it enables over time, an API like this could be seen as a replacement for a hack rather than an introduction of a new capability into the platform.

Yep, and that particular hack has framed a large part of the discussion. But it also showed that in certain situations, boosting brightness wasn't right thing (and that it works in counter-intuitive ways, like making everything else less bright instead).

[beaufortfrancois]

Just boosting brightness unnecessarily or going against the a user's preferences. For example, a site might decide that, "you know what? our site looks super nice when it's on full brightness... we should just ask the user for permission to boost brightness when they view our content because OMG 🤩 so pretty!".

Definitely don't want to end up in the above situation.

@marcoscaceres How would it go against the user's preferences if the web site is actually asking user permission to boost screen brightness? If user says no, the request to increase screen brightness would be rejected.

Note that users can also adjust browser settings to always reject screen brightness requests by default as well.

[HenrikJoreteg]

Hey all. Just had a minor anecdote I wanted to add:

Full brightness is something I first asked for more than 5 years ago as tech lead on the original Starbucks PWA for scanning barcodes at the counter because we noticed even in our own usage that it wouldn't work so consistently with darker screens or in brighter environments. It caught us off guard because we thought it would be fine without it. It caught me at the Starbucks counter several times myself be because my screen was too dark.

If it were not broadly deemed necessary, then why does every single ios and Android app that expects the user to scan a barcode or QR code in real life take advantage of this capability?

I feel like the utility of this functionality is pretty well settled.

Even Google Pay and Apple's own "wallet" apps use this don't they?

I understand the "how" is worth debating, but whether or not it would be useful seems pretty well settled.

Thanks for your time and effort on this everyone.

I'd be so excited to see this land on the web ❤️

[willmorgan]

Hey @marcoscaceres,

For example, the developer could hint to the browser that it wants to take a picture of a person. Then, the browser could bring up the front-facing camera and instruct the user to hold the phone at the right distance away. The user agent could then adjust the brightness by using both the camera's aperture and the screen as a flash, based on lighting conditions if needed.

When what is displayed on the screen is relevant in determining the outcome of a presence check, we're less interested in the ambient light, and more interested in the coloured light that we can control from the screen. We can't rely on the camera's exposure settings to get a clear view and ensure that it isn't an injected video, mask, stick diagram, deepfake, etc. In fact, the "right distance" depends on how much detail and illumination you can get from the user's face from the one-time flashcode.

We could manually calibrate this without touching the screen at all, but that increases the time to completion and detracts from a fast and simple user experience.

There are already applications that can detect if the person is in frame for the purpose of authentication [...] and then knows when your face is in the right place

Absolutely, my use case does parts of this already, here is an excerpt of the scanning experience on iOS today:

RPReplay_Final1658500240.mov

As mentioned before, I would be happy to give you a private and more interactive demo 1:1 and answer any more questions you might have on the spot, any timezone works for me 😁

One could maybe do something like the above also with video embedded in a page?

The above excerpt is done using the video element. However, on iOS, things are even more difficult because:

• there's no way to control front facing camera white balance and exposure time, because advanced MediaStreamTrack constraints aren't supported;
• on iOS, you can only make video elements fullscreen, which means that screen area is further reduced by the address bar, further reducing illumination;
• in any case, using HDR video alone would be problematic for accessibility given we must overlay instructional and status text that is accessible by screen readers to be WCAG AA compliant.

If this is an important use case, I'm asking if we could do better?

IMHO, a screen brightness API is the best and simplest solution for a number of use cases, not just my own. Genuine concerns around user interruption and annoyances can be resolved with the privacy and governance tooling patterns that are already available today, such as permission prompts, permission policy, top level restrictions, gesture tokens, etc.

For my use case, a very tenuous alternative would be using the torch advanced constraint with a facingMode: "user" MediaStreamTrack, and trusting the browser to implement a torch by raising the screen's brightness. However, as above, iOS doesn't support that right now, and it wouldn't solve the QR code scanning use case.

Another real world (not bogus!) use case we're working with, in production today, is performing transactions at an ATM. Being able to adjust the screen brightness only when it's needed means that the screen can be dimmed when the ATM doesn't require it. Otherwise the brightness needs to run at 100% all the time, wasting energy.

In terms of "doing better", and being "webby", I'd argue that the philosophy of the Web has been to provide building blocks for developers to do cool things. Once the community and industry converges on a common usage pattern of building blocks, then it absolutely makes sense to start building the next level of abstraction on top of that.

In this case, I want precise control of the device for a time limited period, so telling a browser "I'm trying to do this complicated thing" and expecting the browser to respond in a potentially vague way might work, but it wouldn't be ideal. I'd rather know whether my brightness request was successful or not, and then act accordingly.

a site might decide that, "you know what? our site looks super nice when it's on full brightness... we should just ask the user for permission to boost brightness when they view our content because OMG 🤩 so pretty!

Two close relations of this API are fullscreen and wake lock; I don't see sites requesting those en masse. Even if they did, the user would need to interact with the site first.

As above, the anti-abuse tooling around this already exists, so in my view... the future could be... bright? 😎

[marcoscaceres]

@beaufortfrancois wrote:

@marcoscaceres How would it go against the user's preferences if the web site is actually asking user permission to boost screen brightness? If user says no, the request to increase screen brightness would be rejected.

Firstly, even allowing the straw-person "brightness for prettiness" situation seems like a non-starter. That shouldn't be possible, and adding the ability for the site to control the brightness for solely that purpose wouldn't be great.

Secondly, that would be adding permission spam: burdening users with a whole bunch of permissions to enable every trivial tiny thing would set an extremely bad precedent. Requesting a permission should have a really high-bar, like with Geolocation, accessing the Camera, etc, where it is really clear as to why one need to enable it. This doesn't feel like it meets that bar, IMO.

We should aim to do brightness without a permission at all, because it's required to perform very specific actions (i.e., why I mentioned automatic detection of QR codes by the browser, for instance). And why also figuring out more sophisticated ways of addressing @willmorgan's use case is relevant. Adding a brightness permission/API feels like a lazy fix, to be honest - but with the serious expense of burdening users with a permission prompt and settings they then need to manage.

[willmorgan]

What about removing the permission prompt and simply requiring a user gesture? Fullscreen works in that way.

[marcoscaceres]

@HenrikJoreteg wrote:

I understand the "how" is worth debating, but whether or not [brightness] would be useful seems pretty well settled.

Yes, I think we are all in agreement about the above. That's a nice way of putting it @HenrikJoreteg.

@willmorgan, I need a bit more time to respond 🙏 ... a lot to process there, but really appreciate all the your input.

[marcoscaceres]

@willmorgan wrote:

What about removing the permission prompt and simply requiring a user gesture? Fullscreen works in that way.

Sort of, yes... but fullscreen has the whole "escape hatch" mechanism around it. Like you enter into it, and the user has full control to break out of it (e.g., hitting esc key). So, it's not as simple as just "click" to enable (that's the easy part)... it's, when does it disengage without having the user do work? specially on mobile where, for example, getting out fullscreen is really cumbersome for anything except video. And when does it only engage when absolutely needed (again, the QR code case or the biometrics case).

[willmorgan]

Sort of, yes... but fullscreen has the whole "escape hatch" mechanism around it. Like you enter into it, and the user has full control to break out of it (e.g., hitting esc key).

This is why tying it to fullscreen would work well, IMHO. It would suit my use case perfectly, and I suppose for QR code cases, that experience would match the iOS wallet perfectly, too.

[marcoscaceres]

Yes, that's effectively:
https://github.com/w3c/screen-wake-lock/blob/gh-pages/brightness-mode-explainer.md#requestfullscreen-integration

(and related proposals to the Fullscreen API)

[HenrikJoreteg]

I'm sorry for letting my frustrations leak out a bit.

I just feel that a different set of rules is consistently being applied to web apps at Apple.

The precedent has been set by Apple that allowing developers control over screen brightness is OK and necessary.

Why does it have to be different for webkit? The ship has already sailed, set on a course chosen by Apple.

[rniwa]

One of the primary benefit of the Web is that it's open (in most jurisdictions). Anyone can make a website, and anyone can visit it; such isn't necessary the case with native platforms. One directive we, as a browser vendor, have to hold to our standards therefore is that users can do so without a fear of infecting their computer with virus, getting their private information stolen, their computers getting damaged, their device's precious battery life depleted, etc... Without these safe guards and guarantees, users will lose confidence in the Web as a place of exploration and, ultimately, the Web will lose out on its biggest advantage over the native platform.

[HenrikJoreteg]

@rniwa in what specific way is a native app fundamentally different to earn this trust and special privilege?

[HenrikJoreteg]

@rniwa you are speaking in generalities, I'd like to hear it explained in this particular scenario.

Whom does this opposition protect? And why do those same users not need the same protection when downloading an app from the app store?

[rniwa]

@HenrikJoreteg : Native apps first need to be downloaded & installed by the user. This is why there are safe guards against running downloaded executables in platforms such as macOS or outright impossible in platforms such as iOS. The web doesn't have that. The Web is a place of exploration. A place where people should be able to experiment freely without having to worry about all sorts of issues that come with running arbitrary executables.

[HenrikJoreteg]

@rniwa I'm not understanding your distinction here. Isn't the user is downloading and executing third party code in both scenarios?

How is the user more protected by downloading third party native code?

[marcoscaceres]

I kindly ask that we keep this discussion thread on topic. @rniwa is absolutely right that the Web is held to a different standard for the reasons @rniwa mentioned (a good thing... they are different platforms). Saying, "well, native apps do it" is not a reason for the Web to do something because the Web is fundamentally different and has a different usage and threat model.

Getting back on the topic of adjusting screen brightness, we suggested some different solutions to the use cases above that might be worth exploring. We are not saying boosting brightness is a bad thing - or that the standards community shouldn't do it.

We are saying: together, let's try figure out a better way of addressing the use cases.

[beaufortfrancois]

Thank you @marcoscaceres for taking the time to provide a position. It is much appreciated.

• Some of the use cases presented could be automatically handled by the UA (e.g., the QR code case).

The "automatic" part may not suit all websites. I'm thinking websites that help users generate QR code don't necessarily want to increase screen brightness. Shall we provide a <meta name="screen-brightness-override" value="none"> tag to indicate when it's not suitable? Detecting its availability would also be nice.

• Allowing arbitrary value adjustments to screen brightness would be open to abuse, or could needlessly drain a device’s battery.

I'm not sure how that's differ from websites running JavaScript intensively, playing HDR videos, and so on... to drain the battery. The UA can decide to reset screen brightness after a certain period of time and even reject requests when it thinks it's not appropriate.

• Relying on .requestFullscreen() would require a change to a web sites, which would be undesirable for developers and potentially inconvenient user experience for users.

What makes you think web developers will find undesirable? In use cases we provided, increasing the screen brightness is ephemeral and the fullscreen experience fits that requirement.

• For other use cases, particularly those requiring biometric authentication, we feel there may be more sophisticated means to address such cases.

Do you mind elaborating on those?

[adrianholovaty]

There's some prior art here with the Web Audio API: audio playback is blocked within web pages unless it's directly in response to user input (e.g., within a click handler).

Could screen brightness possibly use the same approach?

For example, a theoretical changeScreenBrightness() JS API could only be called from within a click handler. Modern browsers should already have the infrastructure to be able to tell which events are user-initiated, because they've implemented it for the Web Audio API.

As a website operator, I'd be comfortable with this limitation.

This goes a long way in addressing the concerns over abuse — which, frankly, I find to be a strawman in any case.

[beaufortfrancois]

There's some prior art here with the Web Audio API: audio playback is blocked within web pages unless it's directly in response to user input (e.g., within a click handler).

Could screen brightness possibly use the same approach?

A user gesture is required indeed. See https://github.com/w3c/screen-wake-lock/blob/gh-pages/brightness-mode-explainer.md#security-considerations.

The screen brightness can be controlled only in response to a user gesture to ensure user retain control over their screen brightness. This prevents a situation where a site increases screen brightness every time the system or user overrides it manually.`

[adrianholovaty]

A user gesture is required indeed.

Thanks for the info! So, in that case...what specifically is the concern about abuse? Seems like it's reasonably buttoned-up.

[tomayac]

The "automatic" part may not suit all websites. I'm thinking websites that help users generate QR code don't necessarily want to increase screen brightness. Shall we provide a <meta name="screen-brightness-override" value="none"> tag to indicate when it's not suitable? Detecting its availability would also be nice.

+1, I share the same concerns. This shouldn't trigger when someone reads https://en.wikipedia.org/wiki/QR_code.

I'm not sure how that's differ from websites running JavaScript intensively, playing HDR videos, and so on... to drain the battery. The UA can decide to reset screen brightness after a certain period of time and even reject requests when it thinks it's not appropriate.

+1. This sounds like a solvable problem.

What makes you think web developers will find undesirable? In use cases we provided, increasing the screen brightness is ephemeral and the fullscreen experience fits that requirement.

Just as a practical side note, iPhone doesn't support requestFullscreen() yet.

[willmorgan]

@marcoscaceres Replying to your various points (on mobile atm):

Battery, abuse concerns etc.

As others have pointed out, such concerns can be dispatched by requiring a user gesture.

Further, in the proposed API, the vendor can choose to opaquely deny a request based on any arbitrary condition - could check the battery, could check OS or browser settings, could deny due to lack of user gesture, and so on.

On requestFullScreen requiring changes from web developers.

AFAICS, the requestFullScreen API wouldn’t suffer any breaking change; it would be a new capability that developers explicitly opt into. In fact it also shares the same error handling with try/catch because the API call can already fail due to missing permissions. Is there some API breakage that the rest of us are missing here?

iOS currently doesn't support fullscreen which means that web developers need to fake it. It's the only user agent I'm aware of that doesn't have this capability, and in fact the lack of this support actually degrades my use case further.

On alternatives, etc.

Could you share your suggestion for more “sophisticated” alternatives? Bearing in mind we have already made numerous iterations on potential API surfaces: DAS WG has produced a number of options: a first-class API, an extension to wake lock, an extension to fullscreen, a CSS property (which the CSS WG sent back)… so I’m at a bit of a loss here.

As always @marcoscaceres if you want a more in depth discussion about what my use case actually does and why it would benefit from this capability, including why it is different from WebAuthn, I am available to go through that.

Ta!

[marcoscaceres]

Thanks again for all the great feedback, questions, and counter arguments, everyone! This is hugely helpful and we will try to draft up responses soon.

Also looking forward to seeing some of you at W3C TPAC where we can also discuss this topic in person.

@willmorgan, yes, let's try to find some time to go through your use cases in the next few weeks. That would be great.

[anssiko]

We have allocated W3C TPAC meeting time w3c/devicesensors-wg#56 for this proposal and @marcoscaceres is invited to this discussion as a guest.

[beaufortfrancois]

Following TPAC Devices and Sensors WG discussion, we've been exploring a declarative approach for allowing web developers to ask the browser to increase the screen brightness.

Please have a look at the proposal.

Update: the proposal link has been updated.

[beaufortfrancois]

As requested at w3c/screen-wake-lock#348 (comment), I've opened WICG/screen-brightness#1 so that @marcoscaceres and others can contribute to the proposal.

[marcoscaceres]

Thanks for moving the discussion over. Will try to find time to continue the discussion there.

[hober]

Closing as we've identified our position.

[beaufortfrancois]

I think we can re-open it as we're discussing a declarative approach with @marcoscaceres.

[hober]

I think we can re-open it as we're discussing a declarative approach with @marcoscaceres.

If there's a new approach, perhaps the best thing to do would be to file a new position request on the new proposal?