Getting Flatpak gamescope working with Flatpak Steam's built in nested sandbox Proton versions. Proton 5.13+ [issue migreted from old utility]

[a73s]

Migrated from flathub/com.valvesoftware.Steam.Utility.gamescope#7

Now that it seems that this package is back from the dead, I would like to bring this long-standing issue to light.

Since Flatpak 1.11.1 beta or later, Flatpak allows for nested sandboxes, like the sandboxes used by Proton 5.13 & Proton 6.3
This gamescope package does not work with these nested sandboxes.

Before the Flatpak 1.11.1 beta, Proton 5.13 & Proton 6.3 would not work with flatpak.
Here is the issue from Steam package: flathub/com.valvesoftware.Steam#642

To get around this issue, the community created flatpak specific versions of Proton & Proton-GE:
com.valvesoftware.Steam.CompatibilityTool.Proton
com.valvesoftware.Steam.CompatibilityTool.Proton-GE

This gamescope package only works with the community made Proton versions, Not the official versions.

The current stable Flatpak release is 1.12.1
With this current Flatpak release, I can install Steam and play like I normally would with a native non-flatpak install.
I can select Proton 6.3 within the Steam GUI, launch games, and they work perfectly in flatpak just like native.
And with the latest versions of MangoHud & vkBasalt they work seamlessly with the standard releases of Proton & Proton-GE as well.
org.freedesktop.Platform.VulkanLayer.MangoHud
org.freedesktop.Platform.VulkanLayer.vkBasalt

However, this seamlessness is not the case with this flatpak gamescope package.
Again it requires the unofficial unsupported community made Proton versions.

It is time this was fixed.
Because it is quite jarring for everything else to work great but then when you try to play that one game that needs gamescope to work better, and it doesn't seamlessly work like everything else you are left there scratching your head ... until you remember you have to install the community Proton build.

This fix could also be useful to other applications using nested sandboxes like HeroicGamesLauncher when it finally gets a flatpak release.

This is still an issue with my testing.

[orowith2os]

This is an issue with Flatpak itself, not this extension; it would be better to make an issue with Flatpak.

per @doraskayo, flathub/com.valvesoftware.Steam.Utility.gamescope#7 (comment)

gamescope launches its own Xwayland instance(s) inside the sandbox and expects games to use them as their X server. However, when the Flatpak portal is used internally by Proton's pressure-vessel to launch a Flatpak subsandbox for a game, Flatpak only exposes the host's X11 socket to the subsandbox, and not the X11 socket created by gamescope's Xwayland instance in the main sandbox. This is why the game window opens as a separate window in the desktop and not inside the gamescope window, and why we effectively lose the "gamescope functionality" as mentioned in this thread with official Proton versions.

The unofficial Proton versions provided in Flathub do not package or use pressure-vessel. In this case there's no subsandbox involved and games have access to gamescope's X11 socket because they run in the same sandbox. This is why using unofficial Proton versions "works around" the issue.

@smcv, flathub/com.valvesoftware.Steam.Utility.gamescope#7 (comment)

Some directories are already shared between Steam and the subsandbox, so there are multiple places where gamescope could create a socket that the subsandbox will be able to see (/tmp and $XDG_RUNTIME_DIR among others), but doing this for /tmp/.X11-unix is complicated by various things:

• Flatpak has to special-case the X11 sockets so that they can be inherited from the host (when permissions allow it, which they have to for Steam)
• Flatpak unconditionally puts a tmpfs on /tmp/.X11-unix to protect unrelated X11 sockets
• I think the way the subsandbox is launched means that the DISPLAY environment variable normally gets inherited from the host, not copied across IPC from the first Steam sandbox

If gamescope could somehow be taught to create its socket in $XDG_RUNTIME_DIR, and just put a symlink to it in /tmp/.X11-unix, then the process that's launched in the subsandbox could be a wrapper that creates a similar symlink in the subsandbox's /tmp/.X11-unix and then execs the actual game? (But the wrapper would have to be either a shell script, a python3 script or a statically-linked binary, because it will be run in a library stack that is not the Flatpak runtime.)

It could be possible for Gamescope to do this workaround, but I'll need to ask a developer about it first.

[misyltoad]

Gamescope has no impact on where the XWayland socket is created fwiu, that's down to the X11 server that's spawned.

I don't think there's a way to move it from /tmp?

[orowith2os]

Ah, yeah, apparently it's hardcoded in the various X11 libraries.... so either this is fixed in Flatpak, or X11/its libraries gets some changes so that it works how we want. The latter should probably work, since that's hopefully where everyone handles it.

[boniek83]

So how to actually make this work with Steam? I have installed com.valvesoftware.Steam.CompatibilityTool.Proton-GE selected it for my game, but nothing happens (ie game just closes) when trying to launch it using:
gamescope -W 3440 -H 1440 -f -- %command%

[jobukkit]

So how to actually make this work with Steam? I have installed com.valvesoftware.Steam.CompatibilityTool.Proton-GE selected it for my game, but nothing happens (ie game just closes) when trying to launch it using:

gamescope -W 3440 -H 1440 -f -- %command%

In the console it just says the gamescope command isn't found even though the gamescope Flatpak is installed.

[poperigby]

I think this part of the README might be relevant:

Here, we include a script that automatically adds the appropriate paths, so all you need to do is add /usr/lib/extensions/vulkan/gamescope/bin to your Flatpak's PATH variable (--env=PATH=$PATH:/usr/lib/extensions/vulkan/gamescope/bin).

I tried this, and it still didn't work though. The game launches and everything, but it just seems to ignore my launch argument.

[JaneSmith]

I can't get either this or the older Gamescope package to work.

I think this part of the README might be relevant:

Here, we include a script that automatically adds the appropriate paths, so all you need to do is add /usr/lib/extensions/vulkan/gamescope/bin to your Flatpak's PATH variable (--env=PATH=$PATH:/usr/lib/extensions/vulkan/gamescope/bin).

I've added this to the Steam Flatpak path. It adds to the PATH correctly, but the gamescope directory seems to be missing even with this package installed:

flatpak run --command=/bin/sh com.valvesoftware.Steam
[📦 com.valvesoftware.Steam ~]$ echo $PATH
/app/bin:/app/utils/bin:/usr/bin:/usr/lib/extensions/vulkan/gamescope/bin
[📦 com.valvesoftware.Steam ~]$ ls /usr/lib/extensions/vulkan/ -al
total 8
drwxr-xr-x. 2 nfsnobody nfsnobody 4096 Jan 1 1970 .
drwxr-xr-x. 4 nfsnobody nfsnobody 4096 Jan 1 1970 ..

[ColinKinloch]

You don't need to edit the flatpak version of steams PATH.
You need the version of the flatpak gamescope package that matches the freedesktop platform steam is currently built against. (currently 23.08, but also install 24.08).
In order to run it with a game under proton you need the flatpak proton version. Currently proton-ge is the only version getting updates so download it and set your game to use it.

Good luck

[JaneSmith]

Thanks for your help. Apologies for my mistake. I only had 24.08 installed; installing 23.08 did the trick. Using Proton-GE, as you said, seems to let Gamescope launch successfully. Thanks!

I noticed that Gamescope still doesn't show up in echo $PATH or command -v gamescope inside the Flatpak, but somehow it does still work anyway when set in the launch args. Interesting. Thanks again.

[devurandom]

I can confirm that installing com.valvesoftware.Steam.CompatibilityTool.Proton-GE and switching the game to it (right click > Properties > Compatibility > Check "Force the use of a specific Steam Play compatibility tool" > Select "GE-Proton9-14 (Flatpak)") allowed Gamescope to do its job and create a working HDR environment for the game. Thanks!

          ID: com.valvesoftware.Steam.CompatibilityTool.Proton-GE
         Ref: runtime/com.valvesoftware.Steam.CompatibilityTool.Proton-GE/x86_64/stable
        Arch: x86_64
      Branch: stable
     Version: 9.14
     License: BSD-3-Clause and LGPL-2.1 and zlib-acknowledgement and Zlib and OFL-1.1 and MIT and MPL-2.0 and LicenseRef-proprietary
      Origin: flathub
  Collection: org.flathub.Stable
Installation: system
   Installed: 1,3 GB

      Commit: fa31066758b80c16dbeee4945118ed1a09e1e830ed2fdc25592baf10cac72006
      Parent: 1b2be4dc01514edc6a14c8b6efd62fd13f247d09660842569e4203143c05da37
     Subject: Update Proton-GE to 9.14 (e04007c7)
        Date: 2024-09-24 16:52:46 +0000

This issue appears to have duplicates:

• gamescope only works with com.valvesoftware.Steam.CompatibilityTool.Proton-GE #138
• CreateSwapchain fails for native linux games in flatpak steam #199

[zastrixarundell]

Just throwing it out there, the issue is mostly with XWayland. With the changes (when it arrives) for Wine/Proton to use Wayland instead of XWayland, would that basically skip over the current issue?

Probably good idea to have XWayland setup for backwards supports for ole' proton versions.

[NastyaGrifon]

Can confirm it's working for com.valvesoftware.Steam.CompatibilityTool.Proton-GE and org.freedesktop.Platform.VulkanLayer.gamescope/x86_64/23.08