Tracking Issue: Gamepad support and device events overhaul

[Osspial]

This issue tracks the implementation of gamepad support and the device events API changes begun in #804. PRs working on this should be made against the gamepad-device-events branch, and we'll merge that branch onto master once it's implemented on all platforms.

• Public API finalized
• Implemented on all platforms
  • Windows
  • macOS
  • iOS
  • Linux X11
  • Linux Wayland
  • WASM
• Updated feature matrix, if new features were added or implemented.

[Osspial]

Mentoring Instructions

At a high level, there are two steps to implementing this:

1. Fix the compilation errors introduced by the device event API changes.
2. Implement gamepad support on your platform of choice.

Completing the first step will involve stubbing out the GamepadHandle functions, and I wouldn't worry about trying to implement them at that stage. These API changes are for the most part aesthetic changes, so fixing them shouldn't involve much fiddling around with the actual platform-specific code. I'd recommend using this step to become more familiar with how the backend works at a high level, and using that to assist with properly implementing gamepad support.

The second step is a bit more tricky, since it involves directly messing with the platform-specific APIs. MacOS and iOS can probably share a significant amount of code for this, since it looks like they use the same APIs for controller input. Similarly, Wayland and X11 can probably both use the evdev API (official page arch linux on Gamepads), but I'm not 100% certain on that. @vberger can confirm or deny that, and give more detailed mentoring instructions on Wayland if he deems necessary.

Enumerating Devices

I've added APIs for enumerating attached MouseIds, KeyboardIds, HidIds, and GamepadHandles, since it's possible (and fairly easy to do) on Windows. I'd be surprised if GamepadHandle wasn't enumerable everywhere, but please bring it up in this thread if it turns out to be impossible to enumerate the other listed types on your platform.

HidId

The HID device API was added mainly because it was easy to do on Windows, but even there it isn't much more than a stub implementation. If you're confused on how to implement this, feel free to just stub it out - we probably need to do some additional API work to make it functional, and I expect there will be a larger discussion on what the API should look like/whether it's necessary.

[francesca64]

I imagine cross-platform rumble is also desired? I don't remember if that was discussed before.

[Osspial]

@francesca64 Yep! That's been implemented on Windows, and is exposed here.

[francesca64]

@Osspial right, but it's only implemented for XInput. The comment I left in for RAWINPUT isn't very helpful, and I learned more after that but never documented my findings.

So, the reality is that XInput is probably the only API that directly exposes rumble to you. Otherwise, you have to go through the platform's low-level HID API and send some device-specific values to the device. The data you send to the device is going to be the same on every platform, so it's not as scary as it sounds.

Chromium's source code is probably the best resource for both finding this info and for finding a reasonable range of devices to support. For instance, here's what you'd generate for the DS4 controller: https://chromium.googlesource.com/chromium/src/+/lkgr/device/gamepad/dualshock4_controller_base.cc#47 - and then WriteOutputReport is implemented on Windows, macOS, and Linux.

From looking at the file list, there actually aren't a ton of devices (I doubt Chromium has sub-par device support, but it could still be a good idea to compare to Firefox, SDL, etc. to be sure). It's still an undertaking to implement, but it seems feasible.

There's also other device-specific functionality out there, like the LEDs on DS4. Unsurprisingly, that doesn't fit neatly into winit's existing API patterns... but the workload is big enough already, so that's something that can be eventually implemented by whomever ends up needing it.

[murarth]

I'm interested in implementing this for Linux X11/Wayland.

I'm currently evaluating options for interacting with the Linux evdev API. So far, the options are:

• Perform ioctl system calls. I believe this would be best done using the nix crate.
  This is the most low-level option. A high-level crate would be preferred, I think.
• The evdev-rs crate; this is a wrapper around the C library libevdev.
  It includes cc and pkg-config among its dependencies. This may complicate the build process for winit in a way that is not acceptable.
• The evdev crate; this is implemented in pure Rust.
  This crate hasn't been updated in 2 years and it has a few significant issues. It would need some work to solve these issues and to modernize the API, but it could be seen as a better starting point than the direct system call approach.

[jansol]

FWIW, I'm in favor of alternative 3 - fixing up the pure Rust evdev crate, as that seems to be the most beneficial solution to the ecosystem overall. Provided that fixing it doesn't require stepping on too many toes of course. Also the repo seems to have been updated this summer so it seems the maintainer hasn't disappeared anywhere but is just busy.

[murarth]

I've been doing some more work on this and I have a little update.

There is an issue with enumerating KeyboardId and MouseId on X11: While the XInput2 API does allow for listing available devices, it does not categorize these devices in a clear manner. Given an XIDeviceInfo instance, one can iterate over the contained classes and search for one of type XIKeyClass; however, there are many non-keyboard devices that have such a class, including Power Button, Video Bus, and USB2.0 VGA Webcam.

X11 could simply present a single virtual KeyboardId and MouseId. This is probably how Wayland would do it, too, as the Wayland client API does not appear to report individual keyboard/mouse devices at all.

As for HID and gamepad devices, both X11 and Wayland can use udev for enumeration, libusb for HID, and one of the aforementioned evdev interfaces for gamepads. (udev could maybe be used to enumerate keyboard/mouse devices, but I'm not sure how we would relate that information back to X11 events.)

It must be noted that udev and evdev are Linux APIs. BSD platforms will need their own interface to these devices and I don't know what that is.

[elinorbgr]

X11 could simply present a single virtual `KeyboardId` and `MouseId`. This is probably how Wayland would do it, too, as the Wayland client API does not appear to report individual keyboard/mouse devices at all.

To be more precise about how wayland does things, in the compositor configuration the user can assign each input device (mouse, keyboard or touchscreen) to a "seat", and the clients only see a single mouse/keyboard/touchscreen per seat. So yes, the Wayland server presents virtual input devices, but there still may be more than one of each kind, and in this case it would be wrong to collapse them into a single device (each wayland seat potentially represents a different user, and each has its own clipboard for example). However, wayland setups with more than one seat are quite niche in general. Though they could typically be used for games doing local multi-player, where each player has a keyboard, all plugged into the same computer, I guess.

[Osspial]

There is an issue with enumerating KeyboardId and MouseId on X11: While the XInput2 API does allow for listing available devices, it does not categorize these devices in a clear manner. Given an XIDeviceInfo instance, one can iterate over the contained classes and search for one of type XIKeyClass; however, there are many non-keyboard devices that have such a class, including Power Button, Video Bus, and USB2.0 VGA Webcam.

I may be misreading the docs here, but could you use the _use field in XIDeviceInfo to get that information? Or does that run into the same pitfalls as using the input device class? Alternatively, could you inspect the keycodes to see if the device is likely to be a keyboard?

If there's really no way to distinguish between the different device types, we could get rid of the distinction between MouseId and KeyboardId. I mainly added that since it was easy to do on Windows, and if that's not portable cross-platform it doesn't make much sense to pretend that it is.

[murarth]

I may be misreading the docs here, but could you use the _use field in XIDeviceInfo to get that information? Or does that run into the same pitfalls as using the input device class? Alternatively, could you inspect the keycodes to see if the device is likely to be a keyboard?

On my system, there is a single virtual keyboard device with a _use value of XIMasterKeyboard. Every other key class device has XISlaveKeyboard, including the non-keyboard devices. Also, every key class device, keyboard or otherwise, has a value of 0 for num_keycodes.

[Osspial]

@murarth I'll go ahead and remove MouseId and KeyboardId's enumerate methods, then. Is there anything else blocking the X11 implementation?

[murarth]

@Osspial I don't think there will be any other major obstacles. I just need to really dig into evdev and get a complete sense of how best to present its data through the winit API.

[FuriouZz]

Hello there! I started to work on the device api changes for wasm32 target.

• Well, I need help to understand how to test too. You do not seem to use wasm-pack.
• It seems that the compilation with the stdweb features does not work =/. Same on the master branch.

92 |     let v = js! { return performance.now(); };
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
   |
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

[goddessfreya]

@ryanisaacg

[ryanisaacg]

@FuriouZz You can use wasm-pack to test the web-sys feature, or you can use cargo web to compile with the stdweb feature.

[FuriouZz]

@ryanisaacg Maybe I am doing wrong, it does not seem to wort =/. I just created a pull request, if you want to look my changes #1414

[FuriouZz]

Any progress about this issue?

[francesca64]

@FuriouZz there's not really anyone around to champion this issue anymore. It's only implemented in 2 backends, and I imagine there's also concern that the needs of the remaining platforms could violate some of the assumptions in the currently planned API design.

I made the original implementation of this for Windows over 2 years ago, so um... at this point, I think we should just release this feature in an incomplete form. It's not useful to anyone if we don't release it, and the missing implementations would encourage people to make contributions that fill in the gaps.

However, I don't have the bandwidth to champion this issue myself. If anyone wants to, getting this to a releasable state doesn't seem too difficult, though there's some cross-platform stubbing and API conformance stuff that needs to be done. The incompleteness would also need to be well-documented.

[FuriouZz]

@francesca64 I can have a look 😊

[MarnixKuijs]

I'm also going to take a look and see if I can help

[jlowry]

Whats the status with this? Does it need more work still? How complex is the work left to be done? I'm asking because I'm considering contributing but would like to understand it before committing to any undertaking.

-- update --
So I've just taken a look at https://gitlab.com/gilrs-project/gilrs and I'm wondering if gamepad support in winit is even nesecary now?

[maroider]

I'm asking because I'm considering contributing

Is there a particular platform you were considering contributing to?

We're currently trying to update the very old gamepad-device-events branch, as master has received a lot of work since gamepad-device-events branched off (see #1856).

[jlowry]

Is there a particular platform you were considering contributing to?

I have access to win10 21H1 , macOs 11.3 on intel and Ubuntu 21.04 and iOs 14.4. I'm not especially interested in supporting any legacy or old versions. I have two wired xbox 360 pads I can test with.

[francesca64]

So I've just taken a look at https://gitlab.com/gilrs-project/gilrs and I'm wondering if gamepad support in winit is even nesecary now?

Motivation discussed here: #695 (comment)

tl;dr it's necessary for robust gamepad support on Windows + having gamepad support on Wayland at all, among other things.

[xgalaxy]

Has anyone discussed supporting SDL2 controller remapping? I know GLFW also supports it.
See: https://github.com/gabomdq/SDL_GameControllerDB

[francesca64]

Has anyone discussed supporting SDL2 controller remapping?

Thanks for bringing this up! It's definitely something we should at least be compatible with. Without it, anything besides xinput is a pain for users.

Parsing/etc. should live in a different crate. In an ideal world, we wouldn't depend on that crate, and users would have an ergonomic way to translate winit events themselves. I'm fine with us integrating it with winit if it turns out we don't live in that world, though.