SDL3 high DPI plan

[slouken]

This is open for discussion of the final high DPI approach for SDL 3.0

There are two basic approaches we've considered:

• SDL uses points for window and mouse coordinates, display scale and actual pixel size are available via separate API functions
• SDL uses pixels for window and mouse coordinates, window and display scale are available via separate API functions

macOS and iOS uses points, Linux and Android use pixels, Windows supports both approaches, selected by the hint SDL_HINT_WINDOWS_DPI_SCALING based on the excellent work by @ericwa.

Using points for window coordinates makes the transition seamless when moving between displays with different scaling. The window size doesn't change, but the effective pixel density changes.

Using pixels for window coordinates is more intuitive for developers and avoids rounding issues when the scale results in fractional pixel coverage (e.g. window width 30 * 1.25 scale = 37.5 pixels)

Other pros and cons for these approaches?

@1bsyl, @flibitijibibo, @icculus, @Kontrabant, @slime73 for discussion.

[slime73]

Android uses pixels

I don't think this is exactly true right now - or at least not in SDL. The Android backend has no DPI-scaling support at all. There's no way to query the scale, no way to use system DPI scaling, and no way to do app DPI scaling without going behind SDL's back. Other platforms at least tend to have system DPI scaling if not the rest.

I think this really needs to be fixed for SDL3. Android should be consistent with the other platforms but it's not.

(In love, we do go behind SDL's back to query the OS dpi scale and convert things appropriately for there, but we shouldn't have to.)

Other pros and cons for these approaches?

I think with pixels-first, it's easy in fullscreen to author an expected/consistent user experience across devices, but in any situation outside of fullscreen it's not easy at all and involves a lot of converting from pixels to points.

With points-first this isn't an issue: things generally have expected sizes across devices, and in fullscreen it's already trivial in a points-first situation to use pixels if you want, as long as the information is available (it's not always available in SDL2.)

For example with pixels-first if I have an 800x600 window, I wouldn't want it to be really tiny on someone's 4k monitor but it will be unless I have a bunch of conversion code. Or if I'm authoring using a 4k monitor I might make at 1600x1200 window and then it'd be really huge on someone's 1080p monitor. This isn't an issue at all with points-first and it's sort of the reason why the concept exists in operating systems.

(This is even aside from multi-display situations, which have similar issues - I just want to illustrate that this sort of problem doesn't need multiple displays, just a pixels-first app that's meant to run on more than the developer's device but doesn't have a lot of conversion code.)

I don't think it's a good development or user experience to have something default to creating sorta-broken apps on different devices. This is what ended up happening in the early era of Windows high-dpi, but macOS completely avoided it because it defaulted to points-first.

[slime73]

That being said, if people aren't convinced then maybe both ways could be prototyped and tested with people's apps on different systems so we can evaluate real situations. (For example even just the SDL test code as-is run in a pixels-first SDL on different systems, and then the diff of it working properly with pixels-first, might be useful.)

If that's done then a lot of the work needed to keep both options at once via a hint would be done too, if that's at all desirable.

[slouken]

We can use SDL_HINT_WINDOWS_DPI_SCALING to try both ways now, on Windows.

[slime73]

I think that would give a partial idea, but Windows is already messy with DPI scaling so it'd be good to test on platforms with more consistency too.

[Kontrabant]

Linux uses pixels

Depends: X11 works similarly to Windows where it operates in pixels and has a font scale value that applications may or may not abide by, while Wayland works in points, so a 4k display at 150% scale has a logical 2560x1440 desktop. The X11 scale value is universal, while Wayland can have per-desktop values.

IMO, points are the way to go for reasons that sloukin and slime73 already went over while adding that, if we do end up going that route, SDL should make it as painless as possible for applications and provide utility functions to handle any necessary points/pixels conversions so that applications don't need to concern themselves with coordinate conversions, worry about how the backend is rounding values, or end up caching scale values, which can cause problems on mixed DPI setups. Things like mouse and window size events can even return both the point and pixel values in the event structure so applications don't have to concern themselves with manually querying them.

Plus, if someone really wants a specific backbuffer size in pixels, they could use a hypothetical pixels-to-points helper function to query the required window point size for their pixel dimensions and resize the window to that.

If that's done then a lot of the work needed to keep both options at once via a hint would be done too, if that's at all desirable.

Personally, I'm not a fan of providing two paths here. Part of the reason why things are a mess now is because there are multiple ways of doing things, and applications often get it wrong. Picking one path that minimizes hazards and provides guarantees that things will work consistently across platforms as long as some basic guidelines are followed would be best and lead to far fewer broken applications going forward, at least in my opinion.

[slime73]

Personally, I'm not a fan of providing two paths here.

I think I agree - but I wanted to throw the idea out there anyway since the decision has been so contentious. 😄

[slouken]

After some discussion with various folks, I think I'm leaning towards going with pixels and display scale factor for everything. This follows the principle of least surprise for existing developers, and gives everyone enough information to implement other methods if they want when wrapping SDL.

This does mean that applications will have to implement the OS provided behavior of automatically resizing windows when moving between displays with different scales, but in the long run that will make for better user experiences (crisp fonts, etc.) and more robust applications.

Let's talk through the other implications of this, so we can make sure everything is covered.

[slime73]

This follows the principle of least surprise for existing developers

Wouldn't this mean every existing SDL developer who made an app would need to add DPI scaling code to their window sizes and positions to make it behave similarly in SDL3 compared to SDL2? And if they don't, there won't be a compile error or whatever telling them what to do, it will just silently be different and the degree of difference will heavily depend on the computer it's running on.

[slouken]

This follows the principle of least surprise for existing developers

Wouldn't this mean every existing SDL developer who made an app would need to add DPI scaling code to their window sizes and positions to make it behave similarly in SDL3 compared to SDL2? And if they don't, there won't be a compile error or whatever telling them what to do, it will just silently be different and the degree of difference will heavily depend on the computer it's running on.

No, Windows and Linux applications will be unchanged, they'll just have a reliable display scaling factor that can be queried. Games which create fullscreen windows will be unchanged. The only applications I think that would be significantly affected are macOS desktop applications that assume 2x scaling.

[slime73]

I don't think I follow - macOS, Wayland, and Windows all use points for window positioning and sizing in SDL right now. SDL's X11 backend currently has no DPI awareness at all so it's the odd one out.

(I believe the X11 backend was also different from Windows even before Eric's Windows work got merged, because unless a developer went behind SDL's back, the OS used points for window sizes and positions on Windows in that situation if I'm remembering right. Also would the same roadblocks that stopped highdpi support in the X11 backend in SDL2 be avoided with this approach? If not, X11 probably shouldn't be considered at all in this decision.)

Also a side note - SDL apps that don't use window positioning, such as mobile apps, may still rely on the point units of a window's size for in-app content scaling and positioning especially because phones have such a wide range of DPI scales in similar form factors. I don't think iOS/Android should be ignored when figuring this out.

[Kontrabant]

Plus, from an implementation perspective, it's much easier and cleaner to adapt pixel based systems to points than vice-versa. Thinking about how to do this in Wayland, and there will be some unavoidable jank with some display configurations, as any window creation on a scaled desktop will need to be immediately resized as Wayland doesn't allow specifying where a window will be created, so it must be created and positioned before the window point size for the desired pixel backbuffer can be calculated. I can also think of a scenario where the window could easily be stuck in a resize loop in a mixed-DPI setup:

1. Partially move a window from a display with 1.0 scaling to a display with 1.5 scaling
2. The desktop tells the window to render with 1.5 scaling so the window point size must be shrunk so the backbuffer is the requested pixel size
3. The window shrinks enough to no longer be crossing onto the second display, so the desktop tells it to render at 1.0
4. The window point size is increased to match the pixel size
5. The resize causes the window cross the display boundary onto the display with 1.5 scaling
6. Goto 2

This could even happen on window creation depending on where the desktop initially positions the window and how much it has to be resized. Wayland doesn't allow programmatically moving windows either, so it would be stuck in this state until the user intervenes.

I have a feeling that other platforms will end up with similar problems, where trying to force point systems to work in pixels will become a battle to make things work and still have failure points.

[1bsyl]

I am not the most familiar with DPI but, maybe we should make this SDL_HINT_WINDOWS_DPI_SCALING be generic for all platforms.

• a developer can set it to 0, to work with pixels
• a developer can set it to 1, to work with DPI
• a user can set it to turn a pixel app, into a DPI one. (or the opposite also ?)

(@slime73 on Android, the only dpi function is: SDL_GetDisplayDPI()
https://github.com/libsdl-org/SDL/blob/main/android-project/app/src/main/java/org/libsdl/app/SDLActivity.java#L1204
https://github.com/libsdl-org/SDL/blob/main/src/core/android/SDL_android.c#L1652 )

[Kontrabant]

I am not the most familiar with DPI but, maybe we should make this SDL_HINT_WINDOWS_DPI_SCALING be generic for all platforms.

* a developer can set it to 0, to work with pixels

* a developer can set it to 1, to work with DPI

* a user can set it to turn a pixel app, into a DPI one. (or the opposite also ?)

Forcing pixel applications to use points can lead to issues one way or another: if it was designed for pixels never assuming that anything but a 1:1 relationship is possible (same as setting SDL_WINDOW_ALLOW_HIGH_DPI and not querying the window pixel size now), rendering and/or pointer coordinates will be broken, or, if those applications are forced to use a 1:1 point:pixel assumption, they will be blurry on scaled desktops (same as not setting SDL_WINDOW_ALLOW_HIGH_DPI now).

In the opposite direction, forcing points to pixels means that things can end up microscopic on high-DPI displays if the application doesn't account for manually handling scaling, and forcing desktops designed around points to work in pixels can lead to some glitchy behavior, such as the example I gave above.

(I believe the X11 backend was also different from Windows even before Eric's Windows work got merged, because unless a developer went behind SDL's back, the OS used points for window sizes and positions on Windows in that situation if I'm remembering right. Also would the same roadblocks that stopped highdpi support in the X11 backend in SDL2 be avoided with this approach? If not, X11 probably shouldn't be considered at all in this decision.)

Yeah, X11 is the odd one out here in that it's not DPI-aware and doesn't do scaling in the way other desktops do. There is a global font scale value that applications can query to see if they should scale UI elements and Xrandr can do some framebuffer scaling, but from an application perspective things will always be 1:1, so it's not really a factor in this discussion.

[slouken]

Okay, you guys have compelling arguments. For SDL 3, we'll continue the move to using points for the window system and have a distinct concept of back buffer size for the window. We'll also provide easy functions to convert from points to pixels for a given window so it's easy for code (like sdl2-compat) to go the other way.

Thanks!

[icculus]

Just so we're talking in specifics, I think this is where we are at the moment. Please correct me if this is wrong or needs changes!

• all things at the window system level (mouse coordinates, window sizes and positions, etc) deal in points, which might be made up of several (and potentially fractional) pixels. This is at the discretion of the operating system, and might be display-specific. It's perfectly legal for 1 point to equal 1 pixel, but it doesn't have to and going forward it will be increasingly unlikely they will be. But since everything will be in points, it shouldn't matter if the window moves to a new display with a different pixel density, it should remain the same visual size to the user and mouse input will still give the same results, etc.

• the only place where we will deal with pixels is at the framebuffer level, which means that completely unrelated to window size, things that deal with framebuffers (SDL_GetWindowFramebuffer for surfaces/software rendering, the 2D Render API, and the GPU API) will talk about the place where rendering occurs in terms of pixels, and either handle this automatically or give the app tools to convert from points. This is the point where programmers have to think about points vs pixels and we handle it for them in some cases, and the rest is hopefully limited to a little bit of app startup code.

• There will be no SDL_WINDOW_ALLOW_HIGHDPI flag. SDL will always attempt to grab as many pixels as it can, regardless of points. This might be contentious, but since this is a question of the OS scaling things or the app scaling things, we should be encouraging apps that care to use render targets to cheaply scale smaller content up, or use the 2D render API's logical size support, if the larger pixel space is going to make things expensive for them. I don't have a lot of pity for people trying to software render to a 4K display.

• Which is to say: most of this already exists in the SDL2 codebase, we just need to tweak a few things and declare these are the official rules, vs the anything-might-happen situation we ended up with in SDL2.

[icculus]

Oh, and: since the framebuffers deal in pixels, if the window moves to a different display with a different pixel density, either SDL or the OS will need to make the existing framebuffer scale as necessary, but the app keeps drawing as it was without having to adjust on the fly. Yes?

[slouken]

Oh, and: since the framebuffers deal in pixels, if the window moves to a different display with a different pixel density, either SDL or the OS will need to make the existing framebuffer scale as necessary, but the app keeps drawing as it was without having to adjust on the fly. Yes?

I think yes, but the application will also get an event it can listen to, to query the new display scale and recreate the framebuffer to match.

[slime73]

For specifics, there were some comments in the old issue (e.g. #2119 (comment) and #2119 (comment)) which had some thoughts/directions for fleshing out high DPI support.

The tl;dr for my comments there is:

• Display modes or displays should have both point and pixel sizes exposed, and/or a DPI scale factor.
• A DPI scale factor for the window might be useful in addition to the current point and pixel dimension query APIs.
• Conversion functions which do OS-specific rounding when converting between the two coordinate systems are useful.
• SDL_GetDisplayDPI is confusing because it's implemented to return physical DPI on many platforms, renaming it would be useful if it's not removed.

There will be no SDL_WINDOW_ALLOW_HIGHDPI flag. SDL will always attempt to grab as many pixels as it can, regardless of points.

Considering the issues we've had with differences in Windows versus the rest (being able to set per-window highdpi on some but not all platforms)... I'm kind of in favour of this. I agree with @ericwa that if we don't remove the use-highdpi APIs entirely then they should be unified into a single thing, at least.

[icculus]

I'm not against any of that, but we also have to deal with the case where the app is running on macOS and is not run from an app bundle that has NSHighResolutionCapable in its Info.plist...I don't know what the state of the art on that is, if we can route around that programmatically, but if we can't, we don't want to get into a situation where the app thinks it has a 150% scaled window but then we get a smaller framebuffer than expected.

(It's also possible none of this is a real concern, I just want to make sure it's not forgotten before we're done.)

[slime73]

I don't think an app is able to query the "true" DPI scale in that situation, so it can't think it has a 150% scaled window there unless it hard-coded that number.

[slime73]

I was thinking last night that just setting the requested mode's DPI scale to the desktop's DPI scale might not be the same as truly matching based on screen coordinates, but now I can't remember if that's true or not.

Ah right, after testing this it isn't the same on macOS - after changing SDL's code to fill in the requested display scale with the desktop's, it makes SDL pick 2880x1800@1x instead of 1440x900@2x when I create a 1440x900 fullscreen window. I think changing the sort order of display modes would "fix" that, but probably isn't desirable in other situations. I believe matching based on the original screen coordinate dimensions instead of the computed pixel dimensions would work in more situations... but overall I'm still leaning toward having less implicit exclusive fullscreen sizing APIs instead.

[Kontrabant]

after changing SDL's code to fill in the requested display scale with the desktop's, it makes SDL pick 2880x1800@1x instead of 1440x900@2x when I create a 1440x900 fullscreen window.

I'm seeing this issue as well when dealing with a scaled desktop and the mode list has two modes with matching pixels, but different screen and scale values:

Mode 0:

• 3840x2160 pixels
• 3840x2160 screen
• 1.0 scale

Mode 1:

• 3200x1800 pixels
• 3200x1800 screen
• 1.0 scale

Mode 2 (the desktop):

• 3840x2160 pixels
• 3072x1728 screen
• 1.25 scale

Even if I explicitly tell the window to use mode 2, the SDL mode matching code wants to use mode 0. This wasn't really an issue before, because the mode list didn't have modes with duplicate pixel values, but now it's selecting something other than what the application expects. The window structure also doesn't communicate that the video core picked a mode for the window different from what the user requested, so the selected mode isn't explicitly passed to the video backend, and the Wayland backend needs that info since the emulated fullscreen mode scaling is handled per-window instead of per-display.

Although, this again begs the question of whether or not we really need the desktop mode in the fullscreen mode list when it has an unscaled duplicate and generally provides no benefit. I understand that it does on Macs, but on other platforms it's no different than just using the display's native 1.0 mode and leads to situations where the application may not get the requested mode if it's too similar to others in the list.

[slouken]

Yes, I think that matching code is wrong. Regardless, I'm going to change the fullscreen mode for the window to be an exact match, rather than doing the closest match that it does now.

[AntTheAlchemist]

In the current source, I have to use SDL_RenderWindowToLogical to convert all mouse coordinates to pixel coordinates so that mouse coordinates match what I'm drawing to the window. ETA on changing mouse coordinates to pixels?

I wish to vote to strip out all OS scaling influence on SDL3. It should always be the app or game that chooses how desktop scaling influences specific UI elements, not the OS forcing global scaling on the whole app to scale up (aka blur) on high density screens.

[slime73]

It should always be the app or game that chooses how desktop scaling influences specific UI elements, not the OS forcing global scaling on the whole app to scale up (aka blur) on high density screens.

I think the only time the OS forced global scaling was in SDL2 when SDL_WINDOW_ALLOW_HIGHDPI and the equivalent Windows high-dpi enable toggle was not set (because operating systems do their own scaling for apps that don't claim to have DPI scaling-aware code.)

Right now in SDL3 that flag is effectively always enabled, so there's no more global blurry OS backbuffer scaling and it's entirely up to your own code to choose what to scale and how to scale it.

This also means you do have to be aware of coordinate systems at some level in your code. Except when fullscreen, your app is one window of many on a user's system and both the user and the system deal in screen (DPI-scaled) coordinates for window sizes and positions. You can convert between coordinate systems but it's not really possible to ignore screen (DPI-scaled) coordinates entirely.

[1bsyl]

@slime73: what's the correct way to set the scale off ? (Does off always means scale == 1.0 ? )

maybe we should add a SDL_SetWindowLogicalScale(). So event comes in the expected intervals. ?

[injust90]

Going with a Pixel system is indisputably better.

[slime73]

Conversion functions which do OS-specific rounding when converting between the two coordinate systems are useful.

Since SDL_Render has some (non-rounding) float conversion APIs with a certain naming convention now, for the above suggestion maybe these names make sense?

int SDL_RoundPixelCoordinatesFromWindow(SDL_Window *window, float window_x, float window_y, float *pixel_x, float *pixel_y);
int SDL_RoundPixelCoordinatesToWindow(SDL_Window *window, float pixel_x, float pixel_y, float *window_x, float *window_y);

I'm still not sure whether they should be window-specific or take a DPI scale float value instead, though. Those names are a bit wordy too.

[slouken]

These shouldn't be rounding. I'll take a look at the Windows case mentioned in #7221.

[slime73]

The rounding function suggestions are for cases like #5827 (comment) - i.e. mostly for making sure someone can achieve the same rounding properties a system backbuffer would have.

I've run into situations where that ended up being important while using Unity (it had a bug where it didn't respect that OS rounding, which broke some things like trying to use a custom depth buffer with the backbuffer). But I'm not totally confident an API suggestion like above is the best way to make sure those sort of problems don't happen. Maybe the existing SDL_GetWindowSizeInPixels covers 99% of that type of use case already.

[slime73]

As for the other thing on my list (SDL_GetWindowDPIScale or a similar name) - in SDL3 now people can do this:

SDL_DisplayMode *mode = SDL_GetCurrentDisplayMode(SDL_GetDisplayForWindow(window));
float scale = mode ? mode->display_scale : 1.0f;

But would a convenience Window function to do that for you still be nice to have and more intuitive? I don't have super strong feelings either way. Here's one reason why the DPI scale of a window's current display can be useful: #5778 (comment)

[slouken]

There's been enough hand-wavey discussion about this that I'd like to have concrete examples of application needs before adding more functions to the API.

[Susko3]

Examples from an application (osu!)

Fullscreen resolution

Users expect resolutions (when using fullscreen) to be in pixels, so I would expect the same from SDL_DisplayMode.

Logical display layout and windows position

In osu!, it's important to know the layout of the displays and the relative position of the current window on that layout (in blue below). This is used for confining the mouse cursor / allowing it to escape when it goes to another display. Some code is here, but it might be confusing out of context. It boils down to checking whether the cursor is inside any display.

The displays that the application gets from SDL should match exactly to the layout set in OS settings. On Windows, display layout uses pixels and not DPI-scaled points. This could be different on other operating systems (macOS is most likely different), so I would expect SDL to provide a cross-platform API on that. And even if an API is not made specifically for this, I would expect SDL_GetDisplayBounds / SDL_DisplayMode to provide sizes in pixels and points, or an easy way to convert. This can allow us to manually choose the right unit for each OS.

[slouken]

Thanks for the example. I think the SDL3 approach will work for osu!
SDL provides both pixels and screen coordinates (points) for display modes, and the monitor bounds are in screen coordinates, so when you lay them out they should have the correct relative positioning. Mouse coordinates and window positions and sizes are in screen coordinates, so they'll be consistent and correct for your confining code.

[Susko3]

As I feared, this doesn't work as expected when tested on latest SDL3 main branch (144390f).

Take this layout for example (the non-100% monitors are both 1920x1080 pixels):

SDL will report it like this if you take the display bounds at face value:

INFO: Number of displays: 3
INFO: Display 2: IPS236
INFO: Bounds: 1280x720 at 0,0
INFO: Usable bounds: 1280x689 at 0,0
INFO:   Desktop mode: 1920x1080@60Hz, 150% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)
INFO:   Fullscreen video modes:
INFO:     Mode 0: 1920x1080@60Hz, 100% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)
INFO: Display 3: SyncMaster
INFO: Bounds: 1200x1600 at -1200,-232
INFO: Usable bounds: 1200x1570 at -1200,-232
INFO:   Desktop mode: 1200x1600@60Hz, 100% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)
INFO:   Fullscreen video modes:
INFO:     Mode 0: 1200x1600@60Hz, 100% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)
INFO: Display 1: PLT2252
INFO: Bounds: 1536x864 at 1920,0
INFO: Usable bounds: 1536x834 at 1920,0
INFO:   Desktop mode: 1920x1080@60Hz, 125% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)
INFO:   Fullscreen video modes:
INFO:     Mode 0: 1920x1080@60Hz, 100% scale, 32 bits-per-pixel (SDL_PIXELFORMAT_RGB888)

To me, this is clearly wrong. And I'm not sure what's the best approach here.

[slime73]

SDL will report it like this if you take the display bounds at face value

This is correct from a UI sizing perspective within each monitor for users - except for the space between monitor 2 and 1. Maybe that's a bug in the Windows backend right now?

Aside from that, you can convert each screen's dimensions from screen coordinates to pixel coordinates just using SDL APIs, if that helps.

[slouken]

I looked more closely at this, and Windows and macOS have a fundamental difference in how they treat monitor placement. As @Susko3 mentioned, Windows works in pixels and macOS works in points, which affects how the mouse travels when it moves from monitor to monitor.

Hmmmmm

[slouken]

The problem is that for monitor layout the two systems are incompatible.

If we have 3 monitors with monitor 1 at 150% scale, the layout for Windows looks like this:

and for macOS looks like this:

Note how monitors 2 and 3 overlap differently with the two layouts. The macOS layout could easily have shifted monitor 1 over to be aligned with the left edge of monitor 2, without any change in Windows layout. There really isn't any mapping from one to the other.

[ericwa]

In the above example, SDL's Windows backend would return the following:

This was the best I could come up with so that the monitor sizes are in dpi-scaled points, but there's still a relatively simple way to convert between Windows native screen coordinate system where everything is in pixels, and SDL's.

Windows itself uses this coordinate system for DPI-unaware apps, and I think Qt does as well?

[slouken]

That seems like a reasonable approach, which would let @Susko3 use the display desktop mode in pixels for layout. The only odd thing here is that they would have to hard-code that behavior for Windows. That's probably fine for this specific case, but are there other cases where this would be problematic?

I think everything works for finding windows and moving them around displays and so forth, because they're all using a consistent coordinate system.

[slouken]

We have updated the SDL 3.0 high DPI support and have come full circle to exposing the native coordinate system and introducing the concept of pixel density and content scale. This is the final plan for SDL 3.0 release.

This was implemented in #7709 and more documentation is available in README-highdpi.md