Mouse position portal

[orowith2os]

Some apps (a small minority, I will admit) have features that pair with being able to track the mouse. One such app is the Virtual Puppet Project's mouse tracker plugin, which uses the mouse instead of a camera to control a virtual character.

What I suggest is exposing a portal that can be used to get the x and y positions of the mouse cursor, rather than showing the surface that the cursor is on to the app. This would allow for having features that use this without giving them root access, and without showing the whole screen.

A plus one would be reporting to the portal the current state of the mouse as well, such as which button is currently being clicked. This would allow for some more legroom in which apps can utilize it.

[orowith2os]

This would compliment the global shortcut portal: #711

[orowith2os]

Hm, this would also probably best be implemented in the InputCapture portal: #714

[orowith2os]

Made an issue in the Wayland Protocols issue tracker, since it's probably not as important to have it be a portal, imo.

https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/110

[GeorgesStavracas]

This would be unsuitable for a portal. Portals are D-Bus based, high level APIs that are not suitable nor designed for high throughput use cases. It probably wouldn't keep up with a 60 Hz stream of cursor position events. This could be achieved by some sort of new mode to the screencast portal where it wouldn't send monitor buffers, only cursor metadata.

But even that wouldn't solve the use case that being presented, which is a program that renders a 3D model whose head keeps pointed at the cursor, because doing that requires knowing the position of the cursor relative to the window, not the absolute position. So such portal wouldn't allow this use case. It also wouldn't work on environments like VR or touch-only environments.

So all in all, I guess I'll reject this portal proposal as is. I think the best way forward is to figure out if wl_pointer could somehow be extended for always receiving motion events in relative coordinated even when the window is unfocused. It already does that with implicit grabs, so it probably wouldn't be crazy to extend from there.

[jadahl]

I think the best way forward is to figure out if wl_pointer could somehow be extended for always receiving motion events in relative coordinated even when the window is unfocused. It already does that with implicit grabs, so it probably wouldn't be crazy to extend from there.

With my Wayland developer hat on, I don't think this is feasible or should be done. It's fine for implicit grabs since they are explicit user interactions and very temporary. Actual mouse tracking by a background app (or an onfocused window, whatever, it's the same in this case) is a very different story, as it in practice can be used to read peoples pin numbers and things like that.

[orowith2os]

Would it be possible to see if a PipeWire video stream/screencast portal could include data about the locations of apps inside of its video stream, or select apps to grab the locations of that are outside of it? It could then drop anything related to capturing visual data too, since it wouldn't be needed in this case (possibly unless requested specifically).

I don't think needing to know the location of one or a specific window's location relative to the cursor is absolutely needed for this, but it definitely would be nice to have, for a complete alternative to xeyes for Wayland.

It going through the screencast portal, or an extension of it, was the idea I settled on in the end, so I don't think worrying about D-Bus being able to handle many events is an issue here.

Here's one more alternative I have in mind: if wl_pointer could be extended (if possible), only allow it to be used outside of the application window if allowed by the user, so it'd go through a portal to have access to that data; but I'd like to consider legacy X applications in this as well, so maybe something in Xwayland would need to be done for that.

I think I'm understanding all of this correctly? I don't have much experience in relation to development like this, just tossing out thoughts at this point.

[jadahl]

Would it be possible to see if a PipeWire video stream/screencast portal could include data about the locations of apps inside of its video stream, or select apps to grab the locations of that are outside of it? It could then drop anything related to capturing visual data too, since it wouldn't be needed in this case (possibly unless requested specifically).

PipeWire is meant as a method for transferring screen content including the mouse cursor sprite and position, it's not meant for introspecting applications etc, so I'd say that's out of scope.

It could then drop anything related to capturing visual data too, since it wouldn't be needed in this case (possibly unless requested specifically).

Whether it should be possible to only get the cursor data of a screen cast, leaving the actual screen capture pixels be is a question for PipeWire, not the portal.

Here's one more alternative I have in mind: if wl_pointer could be extended (if possible), only allow it to be used outside of the application window if allowed by the user, so it'd go through a portal to have access to that data; but I'd like to consider legacy X applications in this as well, so maybe something in Xwayland would need to be done for that.

I don't think using wl_pointer for this makes much sense, it's an object that already has a clear purpose, and implementations are likely to have clear focus management code that decides how to route events, and it doesn't really make anything easier using it directly compared to inventing something completely new.

Hm, this would also probably best be implemented in the InputCapture portal: #714

The input portal as it looks now are for capturing the input, i.e. it's meant that input either go to the capturer or the display server, not both at the same time. The reason for this is to not allow e.g. keyboard loggers (which mouse cursor position trackers are quite similar to).

[orowith2os]

Wanted to bring this up one more time, since I had an idea that might fit the fact that dbus isn't meant for high throughput tasks:

A Wayland protocol is most likely out of the question, but there's another method that could be used for fast information transfer: the filesystem. Applications can communicate with the compositor constantly using UNIX domain sockets, why couldn't a portal initialize something like that and applications read the appropriate file for mouse information?

Being able to tell the relative position of the mouse also might not be a worry, as an app could easily figure out where it is on the X,Y plane by viewing the mouse events and the Wayland events if needed.

[orowith2os]

Thinking about this again (with some discussion with some friends), this could be managed by an accessibility interface. Android, for example, allows apps to ask the user if they would like to allow an app privileged access to the windows, such as for emulating or logging input. I wonder how feasible it would be to port many of Android's accessibility features to Linux, especially as a portal to make it more sandbox friendly.