[Question] Opening the on-screen keyboard with standard APIs

[flibitijibibo]

Currently Valve asks applications to interact with the on-screen keyboard via proprietary APIs, such as ShowGamepadTextInput - this is okay for new games, but for older games there is already prior art for interacting with the on-screen keyboard, namely SDL's text input API. On Wayland we actually have really good on-screen keyboard support thanks to the text-input-v3 protocol, which does a great job of not only showing the keyboard but only doing so when a physical keyboard isn't present - once gamescope supports native Wayland clients (#543) this will all hopefully Just Work for many existing native SDL titles!

For now though, we're of course using Xwayland, and SDL's X11 driver does have support for ibus/fcitx, but I'm not aware of any standard way to interact with the on-screen keyboard - does gamescope/wlroots expose this to X applications in some way? If we can add this to SDL, it would be a huge help for existing games; I've lost count of how many games I've had completely fail Deck cert because SDL_StartTextInput doesn't work on Steam Deck, even though it works great on normal Linux systems, and unfortunately updating Steamworks is not as trivial as updating SDL.

For reference, here's SDL's...

• X11 keyboard file https://github.com/libsdl-org/SDL/blob/main/src/video/x11/SDL_x11keyboard.c
• Linux core folder, which includes the IME files https://github.com/libsdl-org/SDL/tree/main/src/core/linux

[misyltoad]

Hi!

Gamescope has nothing to do with the Steam keyboard -- that's provided entirely by Steam.

There are two possible options we could do to make this work:

Option 1

SDL gets the GAMESCOPE_WAYLAND_DISPLAY, and uses its existing text_input_v3 interface to talk to Gamescope, which talks to Steam to open the keyboard.

We do technically support Wayland clients -- just not xdg-shell after all.

Option 2

SDL detects its running on Gamescope by checking the root window of the DISPLAY.

SDL uses xdg-open or whatever and just calls steam://open/keyboard?XPosition=%d&YPosition=%d&Width=%d&Height=%d&Mode=0 to open the keyboard

Some games on Steam that are bound by older Steam APIs as they use crappy middleware, etc end up doing this.

---

Maybe @slouken has an opinion on what method they would like in SDL?

[flibitijibibo]

Option 2 would probably be the best route for the x11 driver - we could implement HasScreenKeyboard to check for gamescope, then StartTextInput could call the deeplink. We would probably just use whatever values make the equivalent of pressing Steam+X, if possible.

[misyltoad]

Option 2 needs nothing from our side so feel free to give it a shot if you think thats the best.

[flibitijibibo]

I'll throw together a draft today - for Wayland clients we should definitely use the official protocol, but that can come from gamescope later IMO (SDL3?).

[slouken]

Option 2 is also extendable to other environments, so I would vote for that as well.

[slouken]

I would implement it via the ScreenKeyboard internal API, and call SDL_OpenURL() to open the on-screen keyboard. Maybe this is appropriate for src/core/steam/SDL_steam.c?

[slouken]

We probably want to account for the case where there's a hardware keyboard attached?

[misyltoad]

We probably want to account for the case where there's a hardware keyboard attached?

Steam should handle that in the open/keyboard call.

[flibitijibibo]

libsdl-org/SDL#6515

[flibitijibibo]

This can now be seen in action with Salt & Sanctuary's native Linux version (which is currently being overridden by force, grumble):

https://steamcommunity.com/games/SaltandSanctuary/announcements/detail/5023230335822734611