scale the local display

[totaam]

Issue migrated from trac ticket # 976

component: client | priority: critical | resolution: fixed

2015-09-04 08:52:23: antoine created the issue

---

Allow the client to "lie" about its real display dimensions (and maybe still report the actual correct values somewhere) so that the server can render things at a different resolution and save CPU, memory and bandwidth.

This will allow us to handle 4k screens a lot better: we can render at 2k or even just 1080p and upscale client side.
For a lot of content, especially video-like pixels, it is a lot more efficient than upscaling the video (in the client application: browser or video player) then downscaling it before compressing it, only to upscale it at the other end. The compression can introduce a lot of blurring which gets magnified. Compressing 4k is just too expensive in most cases, even with hardware assisted encoding.
With a smaller virtual screen size, we are more likely to be able to compress losslessly or at a better quality and higher framerate.
The client-side upscaling may also be able to do the anti-aliasing for us.
This is what MS Windows does already for applications which are not DPI aware.

Maybe worth doing if dummy + randr does not get fixed in time: when we are scaling things, we could also use the full virtual screen's size (which may be bigger than the size the client requested) and let the client scale it appropriately. That's as long as the differences are minimal (say 10%?). It will avoid bugs with applications which use the display size without checking the desktop / workarea / available size options.

Related / impacted tickets:

• DPI: pass client DPI preference to server (and use sane defaults of 96dpi) #163, gtk screen dimension detection is broken with high dpi displays on windows7 and later #697, gtk is confused about some win32 screen settings #698
• dealing with very large client screen sizes #969, if an application changes the screen size on the server via randr - what do we do? #33, FullScreen mode in xpra client #972
• allow client side shadow windows to be resized #530
• opengl rendering: opengl errors with very large windows #942,
• dummy / randr: xrandr reports screen dimensions as 0mm x 0mm #728, some parts of windows can not be clicked after changing resolutions #349, add new resolutions to Xdummy on the fly - dummy driver needs randr support added? #56, browser failing to render to actual size #968
• shadow servers: generic shadow improvements #899

[totaam]

2015-09-04 18:27:35: antoine uploaded file client-scaling.patch (41.0 KiB)

work in progress - rendering part not done (big)

[totaam]

2015-09-04 18:28:22: antoine changed status from new to assigned

[totaam]

2015-09-04 18:28:22: antoine commented

---

From the patch above, things that need to be fixed or added:

• session info updates: show scaling somewhere, adjust values
• expose to server? (so we can debug via xpra info)
• get_frame_extents / get_window_frame_sizes / get_window_frame_size - not done
• cursors (not scaled) - not sure if we should..
• downscaling client side is wasteful: the server should downscale instead, disallow values lower than 1.0?
• pixmap and cairo backings not done: keep track of render size and scale as needed (using Pillow, gdk pixbuf and csc...)
• resize support: could scale until the next screen update comes through
• anti-aliasing tweaks?
• tray menu entry to control?
• magic keys bindings to upscale / downscale individual windows, control channel? (should probably affect all windows for a group leader?)

to verify:

• get_client_window_classes could use backing size? the opengl limit is on the texture? (but we also render to a PBO?)
• geometry hints handling
• system trays: maybe those should not be scaled in size, just position?
• spinners
• toggling opengl on / off
• alpha, shape, etc..

[totaam]

2015-09-05 08:56:35: antoine uploaded file client-scaling-v2.patch (43.5 KiB)

mostly working patch - opengl done

[totaam]

2015-09-07 06:38:32: antoine uploaded file client-scaling-v6.patch (53.2 KiB)

working with all backends: cairo, pixmap and opengl!

[totaam]

2015-09-07 07:17:26: antoine commented

---

Mostly done in r10533, currently only accessible via env vars. It looks really good and uses a lot less bandwidth! (much better than the automatic scaling - which may still kick in, but much less)

Note: the pixmap and cairo backings look quite ugly compared to the opengl one because we only repaint the area that has changed whereas the opengl backend repaints everything (because this is cheap) and therefore avoids aliasing issues on the edge of the area being repainted. We could update those backends I guess. Needs testing to decide if it is worth it.
The Pixmap backend upscales the picture using the "HYPER" gdk filter, this can be overriden with XPRA_SCALING_INTERPOLATION=FILTERNAME - see here for the list of options: pixbuf.scale.

Still TODO:

• most items from comment:1
• maybe support specifying the scaling as ratios, as we already do server side via parse_scaling_value (ie: support 3/2), and send this to the server instead of a float multiplied by 1000...
• add command line switch: --client-scaling=off|auto|XVALUE,YVALUE, auto could enabled client side scaling if DPI is high? or if the desktop size is very big, or both?
• maybe update opengl errors with very large windows #942: we now can have windows bigger than the max texture size by downscaling them... and maybe we should automatically downscale in this case? (and check for GL_MAX_VIEWPORT_DIMS instead - which should always be bigger than the texture size already?)
• maybe implement FullScreen mode in xpra client #972 the easy way for now: don't deal with offsets, just scale the window to fullscreen size
• spinners with opengl are probably using the wrong dimensions?
• verify the DPI issues with win 8.1 and later: we really don't want the OS to upscale our window contents again if we've done it ourselves (scaling is often a lossy operation unless we're multiplying the size by a round number, so best to do it just once)

[totaam]

2015-09-08 15:30:25: antoine commented

---

Important fixup in r10553 - unscaled opengl rendering was broken since r10533 on platforms without double-buffering (all but win32).

[totaam]

2015-09-11 19:30:04: antoine commented

---

• r10591 renamed the "scaling" option to "video-scaling" to prevent confusion, still controls video scaling aggressiveness (this scaling happens server side, before the video encoding)
• r10621 adds the "desktop-scaling" option, see below for examples
• r10622 adds the "auto" mode, which will automatically enable scaling for client resolutions above 1080p: 150% up to WQXGA, 200% up to UHD, 300% for FUHD and 400% above that (see wikipedia: Graphics display resolution for the definition of the acronyms)
• r10624 ensures we don't select scaling values too low, which the server would not be able to handle - also enables scaling automatically if the server max-size is too small to handle our screen size (as long as scaling is enabled obviously)

Here are some usage examples:

• double the size of all windows: desktop-scaling = 2
• increase the size by 50%%: desktop-scaling # 1.5 or desktop-scaling3/2
• disable all desktop scaling: desktop-scaling # off or desktop-scaling0
• enable desktop scaling, but start with no scaling activated (the default): desktop-scaling # 1 or desktop-scalingon
• auto mode: desktop-scaling=auto

Still TODO:

• man page updates
• blurriness seen in partial repaints with cairo and pixmap backings (disable scaling with those, or use slow-full-window repaint?)
• dpi and cursors?
• test system trays?
• limit downscaling more? (useful for testing, but a complete waste of bandwidth and CPU)
• xshape fixes..

(some of these items are likely to be scheduled for a future release, ie: xshape)

[totaam]

2015-09-16 10:08:33: antoine changed status from assigned to new

[totaam]

2015-09-16 10:08:33: antoine changed owner from antoine to afarr

[totaam]

2015-09-16 10:08:33: antoine commented

---

• man page updates in r10647
• scaling cursors done in r10648
• dpi should be ok as it is.. no better or worse I think, though the new virtual size does make it harder to figure out what the "real" dpi should be!
• systray forwarding fixed in r10650
• not going to limit downscaling: let people do stupid things, can also be useful to display a large desktop (ie: 4k or above) on a very small screen (ie: vga)
• xshape: fixed in r10652 (not perfect - not sure we can make it much better, and not sure we care)
• r10653 adds a keyboard shortcut to reset scaling to 1: Alt+Shift+KP_Multiply (aka "*" on numpad)
• r10655 fixes repaints with non-opengl backends: we now repaint the whole window when scaling (see commit message for env var to tweak this)

---

@afarr: ready for testing:

• the new command line switch, including "auto" mode which we may want to use as default in the future (and we can adjust the thresholds if needed)
• key shortcuts to toggle at runtime: Alt+Shift+Plus and Alt+Shift+Minus, and ensuring we don't overflow the max server screen size
• system tray forwarding (ie: vlc has one)
• dpi - not sure how to test?
• cursors should scale up / down too
• xshape (ie: xeyes uses that)

---

future stuff (recording here for lack of a better place):

• add a tray menu to easily toggle scaling?
• add server-side downscaling? (would save lots of bandwidth if the client is not going to use all those pixels)

[totaam]

2015-09-18 22:28:02: maxmylyn commented

---

Running a Fedora 20 r10666 client against a Fedora 21 r10666 Server:

• Tested --desktop-scaling=auto (unable to trigger due to resolution being at 1080p), everything works the same as before, even after a couple hours of normal usage (surfing Trac comments in Firefox, emails in Thunderbird).

• Key shortcuts work(as in manually scale) as expected.
• System tray forwarding does not work properly

• The icon stays the same, but when scaling down or up, the right click menu on VLC no longer works

• Cursor scales down, but stays small when resetting scale (shift + alt + * (even sticks after a reconnect)

• Edit: The cursor does not scale up properly as well

• Not sure how to test xshape

[totaam]

2015-09-19 05:48:29: antoine changed owner from afarr to maxmylyn

[totaam]

2015-09-19 05:48:29: antoine commented

---

System tray forwarding does not work properly
[[BR]]
Which desktop environment do you use?
With gnome3 and topicons, the tray icons don't work half the time - that's even without xpra...

And even without the topicons extension, the horrible and annoying little widget at the bottom left hand side of the screen doesn't work either!

I'll take a look later since I also get problems with win32 (and that one works reliably since Windows 2000!)

Some minor fixes in this area in r10667. (should help when upscaling)

[[BR]]

Cursor scales down, but stays small when resetting scale (shift + alt + * (even sticks after a reconnect)
Edit: The cursor does not scale up properly as well
[[BR]]
Fixes to upscaling cursors in r10668.

Notes:

• there's a limit to how big we can make the cursor (128x128 on my dev box) - and the limit calculations were wrong
• on many platforms (X11 for a start), the cursor won't actually change until you move the mouse by at least one pixel
• some platforms (like win32) use fixed cursor sizes (we could add code to paste the cursor into a transparent frame to emulate smaller sizes - meh, the fixed size is quite small already)
• for testing, this little test script is useful: [/browser/xpra/trunk/src/tests/xpra/test_apps/test_custom_cursor.py ./tests/xpra/test_apps/test_custom_cursor.py] (each window uses a different type of cursor - see window title)
• if you still have problems, please include -d cursor debug output.

[[BR]]

Not sure how to test xshape
[[BR]]
xeyes

[totaam]

2015-10-08 15:47:58: antoine changed priority from major to critical

[totaam]

2015-10-08 15:47:58: antoine commented

---

raising the priority: this is one of the most important new features in 0.16, and I would like to make "auto" mode the default

[totaam]

2015-10-08 22:44:11: maxmylyn commented

---

The client(fedora 20) is currently running Mate, but I have Gnome, and Xfce installed - and use them from time to time.

Upped server (Fedora 21) and client (fedora 20) to trunk r10765:

• Cursor no longer scales for me (up or down)
• thanks for the python file - that helps

-d cursor output(connected, scaled a couple times, disconnected):

2015-10-08 14:33:13,534 Attached to tcp:10.0.32.139:2200 (press Control-C to detach)

2015-10-08 14:33:13,891 window dimensions are wrong: 1x1
2015-10-08 14:33:13,926 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
/usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
  import gobject._gobject
2015-10-08 14:33:15,622 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:15,623 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:15,624 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9d00: GDK_LEFT_PTR>
2015-10-08 14:33:15,664 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:15,665 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:15,665 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9b48: GDK_LEFT_PTR>
2015-10-08 14:33:15,875 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:15,876 setting new cursor by name: hand2=<enum GDK_HAND2 of type GdkCursorType>
2015-10-08 14:33:15,877 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9b48: GDK_HAND2>
2015-10-08 14:33:15,979 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:15,980 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:15,981 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9d00: GDK_LEFT_PTR>
2015-10-08 14:33:16,775 sound-sink using audio codec: MPEG 1 Audio, Layer 3 (MP3)
2015-10-08 14:33:18,032 sending updated screen size to server: 3840x2160 with 1 screens
2015-10-08 14:33:18,033   :0.0 (602x343 mm - DPI: 162x159) workarea: 3840x2064 at 0x48
2015-10-08 14:33:18,035     DVI-I-1 (600x340 mm - DPI: 162x161)
2015-10-08 14:33:18,394 scaling_changed() resetting cursors for: [ClientWindow(18)]
2015-10-08 14:33:18,395 set_windows_cursor([ClientWindow(18)], ..)
2015-10-08 14:33:18,396 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:18,397 make_cursor(..)=<gtk.gdk.Cursor at 0x3efc238: GDK_LEFT_PTR>
2015-10-08 14:33:19,387 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:19,435 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:19,438 setting new cursor by name: xterm=<enum GDK_XTERM of type GdkCursorType>
2015-10-08 14:33:19,439 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9df0: GDK_XTERM>
2015-10-08 14:33:20,346 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:20,347 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:20,348 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9d28: GDK_LEFT_PTR>
2015-10-08 14:33:20,564 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:20,565 setting new cursor by name: hand2=<enum GDK_HAND2 of type GdkCursorType>
2015-10-08 14:33:20,566 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9d28: GDK_HAND2>
2015-10-08 14:33:20,603 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:20,605 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:20,606 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9cd8: GDK_LEFT_PTR>
2015-10-08 14:33:21,613 set_windows_cursor([ClientWindow(1), ClientWindow(40), ClientWindow(10), ClientWindow(11), ClientWindow(12), ClientWindow(18)], ..)
2015-10-08 14:33:21,614 setting new cursor by name: left_ptr=<enum GDK_LEFT_PTR of type GdkCursorType>
2015-10-08 14:33:21,615 make_cursor(..)=<gtk.gdk.Cursor at 0x3cf9df0: GDK_LEFT_PTR>

[totaam]

2015-10-09 11:35:09: antoine commented

---

Ah, right, sorry about that, for testing I had been running my client with:

XPRA_USE_LOCAL_CURSORS=0 xpra ...

Which disabled named cursors, r10781 ensures we never use named cursors when scaling (as named cursors cannot be resized).

I'm still seeing cursors that seem too big when scaling or when using XPRA_USE_LOCAL_CURSORS=0, but I think that's just a different server configuration.
If everything else works, please re-assign to me and I'll take another look.

[totaam]

2015-10-09 20:57:15: maxmylyn changed owner from maxmylyn to antoine

[totaam]

2015-10-09 20:57:15: maxmylyn commented

---

Upped to r10783:

• Fedora client now scaled the cursor properly.

• Windows client does not scale properly (not sure if it's expected or not)

• Scaling down then attempting to use VLC's tray icon in Windows causes the right click to be sent to Google-Chrome (?) - and then Chrome's right click menu is activated

• Not sure what logs you'd like, let me know and I'll provide

• Resetting scale causes it to work normally

• Scaling up/down in Fedora 19 causes VLC's tray icon to stop working altogether

---

Here's the -d cursor output for a quick connect-and-scale session:
[/attachment/ticket/976/976-comment11-cursor-log.txt]

[totaam]

2015-10-10 06:44:17: antoine uploaded file 976-comment11-cursor-log.txt (16.6 KiB)

moving the cursor log to an attachment

[totaam]

2015-10-10 07:08:10: antoine changed status from new to assigned

[totaam]

2015-10-10 07:08:10: antoine commented

---

Windows client does not scale properly (not sure if it's expected or not)
[[BR]]
It is.

The situation on win32 is unclear, see #998.
Cursors are meant to be using a fixed size so for the time being we do this:

• if the cursor is meant to be smaller, we paste the cursor image onto the target size (leaving the edges transparent), so it looks the right size (so downscaling should sort of work)
• we can't make the cursors bigger than the fixed cursor size, so we just scale them up (or down) to that maximum size... shows up as:

scaling cursor from 48x48 to fixed OS size 32x32

[[BR]]

Scaling down then attempting to use VLC's tray icon in Windows causes the right click to be sent to Google-Chrome (?)...
[[BR]]
That's because we're not adjusting the tray icon's location properly to its new server-side on-virtual-screen position (scaled).
And so the clicks end up landing somewhere else...
Will fix.

[totaam]

2015-10-11 17:48:43: antoine changed status from assigned to new

[totaam]

2015-10-11 17:48:43: antoine changed owner from antoine to maxmylyn

[totaam]

2015-10-11 17:48:43: antoine commented

---

As of r10798 we now paint the monitors and the workarea bounds onto the virtual screen with sync-xvfb (see #988), and the tray seems to be in the right place.

Turns out that this HUGE bug was not tray related but only more apparent with the systray, and it would have caused us all sorts of problems elsewhere - now fixed in r10802. Works for me with win32 and vlc.

PS: I'm seeing some visual corruption with win32 when dowscaling without opengl. (will try to deal with it later - should be good enough for testing as it is)

[totaam]

2015-10-13 20:06:33: maxmylyn changed owner from maxmylyn to antoine

[totaam]

2015-10-13 20:06:33: maxmylyn commented

---

Upped to r10786 Server and r10810 Client (Installed from xpra.org/beta):

• VLC tray works as expected in Windows

• I have also seen mild visual corruption on the taskbar icon when scaling up and down

• Tested Fedora and Windows clients, mild corruption only notable in Windows(you can see the quality change on Fedora, but that's all)

---

Since everything's working, I'll pass this back to you to decide what else needs to be done.

[totaam]

2015-10-14 05:45:48: antoine changed owner from antoine to afarr

[totaam]

2015-10-14 05:45:48: antoine commented

---

taskbar icon when scaling up and down... mild corruption only notable in Windows
[[BR]]
Yes, I had noticed that - it looks as if it's painting the new one on top without clearing the previous one, but I have no idea where that is coming from - so we will have to live with it for now.

We already have a test app we can use to verify that changing the tray icon works and it's running fine with Windows clients: [/browser/xpra/trunk/src/tests/xpra/test_apps/test_system_tray.py] (just click on the tray icon to change to the next one)

---

Since this works well enough, I have made "auto" mode the default in r10831.
This is a big change.

@afarr / maxmylyn: this will need testing with 4k monitors (and multiple monitors, etc), especially with different DPI settings (see #163 / Writing DPI-Aware Desktop and Win32 Applications).

Windows 8.1 onwards used to scale our windows automatically on high dpi displays (see DPI and the Desktop Scaling Factor), but as of r10832 we should now claim to be Process_System_DPI_Aware via SetProcessDpiAwareness. You can use XPRA_DPI_AWARENESS=VALUE to change this value (Process_System_DPI_Aware # 1, Process_DPI_Unaware0, Process_Per_Monitor_DPI_Aware = 2 - I don't think we will be able to handle per monitor DPI awareness for a while without some major changes X11 side). Use -d screen to see the DPI calls, which should happen very early in the client startup process.

Note: I have no way to test this as I don't have a non-virtualized Windows 8.1 or later system to test on.

Important: without the "auto" desktop scaling mode added in r10831, we may end up using a lot more bandwidth with r10832 on Windows 8.1 and later since we should then be using "actual" monitor resolution instead of the DPI downscaled one. (a lot more pixels to forward)
Why do we want to do this ourselves? We can scale more than the OS and we will get better quality if things are scaled only once by our client.

Maybe we should be taking into account the desired DPI to decide when to scale instead of just the display resolution?
(if the user or OS chooses a high DPI, then we may want to scale - using GetDpiForMonitor, Determining the DPI Scale Factor)

Maybe also re-test #919 at the same time as we may have to catch WM_DPICHANGED events (this page has a nice summary table listing scaling values for each DPI range) and check to see if the values for the frame extents have changed.

Note: I don't think we should be using the same scaling values as what MS Windows uses: scaling more is good for us as we save more CPU, bandwidth and we get a better framerate and user density.

[totaam]

2016-01-13 23:05:23: afarr commented

---

Ok — tested with 0.17.0 r11687 osx and windows clients against 0.17.0 r11692 fedora 23 server.

Percentages work, decimals work, "relative" scaling works & is maintained as scaling is shortcut-adjusted up or down... mixing percentages and decimals works.

Then I remembered to try the shift-alt-asterisk shortcut with scaling at something odd (129%x150%?; scaled down from initial 143%x166%) ... and noticed that it reset to a simple 100% scaling (rather than the "initial" setting)... and from there it scaled up as expected (125%, 150%, etc.).

I'm not sure if the asterisk short cut should go back to 100% or to the initial setting. I could see the virtues of leaving it as is for users to "turn off" scaling... or have it jump back to the initialized default. The "turn off" option has the distinct appeal of allowing us to close this ticket.

I'll hand this back to you yet again to decide.

[totaam]

2016-01-21 06:23:01: antoine changed owner from antoine to afarr

[totaam]

2016-01-21 06:23:01: antoine commented

---

• r11703 will restore the initial scaling rather than 100%,100%
• r11704 adds a shortcut for turning scaling off (100%, 100%): Meta+Shift+question

@afarr: if that works for you then please close. (I'm not going to backport these)

[totaam]

2016-01-22 01:02:43: antoine commented

---

r11718 fixes the scalingoff shortcut (missed commit on window class), beta win32 build uploaded.

[totaam]

2016-01-22 01:37:17: afarr changed status from new to closed

[totaam]

2016-01-22 01:37:17: afarr set resolution to fixed

[totaam]

2016-01-22 01:37:17: afarr commented

---

Testing with win32 0.17.0 r11718 client against a 0.17.0 r11705 fedora 23 server... all those last fixes are working as expected.

Since I can't think of any other nitpicks, or find anything else - I'll close.

[totaam]

2016-01-28 06:16:15: antoine commented

---

Follow up in #1101