Wayland / PipeWire capture

[GeorgesStavracas]

Description

Add a new Linux capture based on PipeWire [1] and the Desktop portal [2].

This new capture starts by asking the Desktop portal for a screencapture session. There are quite a few D-Bus calls involved in this, but the key points are:

1. A connection to org.freedesktop.portal.ScreenCast is estabilished, and the available cursor modes are updated.

2. CreateSession() is called. This is the first step of the negotiation.

3. SelectSources() is called. This is when a system dialog pops up asking the user to either select a monitor (desktop capture) or a window (window capture).

4. Start() is called. This signals the compositor that it can setup a PipeWire stream, and start sending buffers.

The reply to this fourth call gives OBS Studio the PipeWire fd, and the id of the PipeWire node where the buffers are being sent to. This allows creating a consumer PipeWire stream, and receive the buffers.

Metadata cursor is always preferred, but on the lack of it, we ask the stream for an embedded cursor (i.e. the cursor is drawn at the buffer, and OBS Studio has no control over it.)

Window capturing is implemented as a crop operation on the buffer. Compositors can send big buffers, and a crop rectangle, and this is used to paint a subregion of the buffer in the scene.

The new capture is only loaded when running on EGL, since it depends on EGL to call gs_texture_create_from_dmabuf().

Motivation and Context

OBS Studio recently gained support for running as a native Wayland client, but there is no capture yet. Capturing the desktop or windows is just as important.

How Has This Been Tested?

On a Linux + Wayland session, run OBS Studio and add either the "Screen Capture (PipeWire)" source, or the "Window Capture (PipeWire)" source.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[GeorgesStavracas]

This is very early work; I'll continue iterating on this PR throughout the next few weeks. Leaving it as a Draft until then.

[kkartaltepe]

The only issue I have with this, is that the API is entirely focused around making zoom/hangouts/whatever work in your browser. Normal OBS user workflows such a scene per window/application are currently unimplementable. Users who change scene collections get a wave of permission and scene selection prompts. You cant just have OBS work if you are opening and closing windows, because once you lose the capture you need to open OBS and restart your capture. Finally if you tried to setup multiple captures on some platforms, such as the most widely supported, xdg-desktop-portal-gtk you will encounter weird bugs such as unclickable dialogs (flatpak/xdg-desktop-portal-gtk#334 the rate of activity upstream is also slightly concerning since our users will depend on them to use OBS on wayland in this configuration). At the API level there is also no metadata about the request, so when prompts do come up users cannot tell which source they are even selecting the window/monitor for.

Current usability issues aside, if upstream was willing to consider changes to support actual OBS workflows it would be nice. It seems premature to merge support without even discussing with upstream about these changes. Users will understandably consider OBS workflows perfect the moment someone on phoronix claims wayland is supported, and we dont want a wave of unhappy users when we can collaborate to fix the issue beforehand. And we will be stuck with it once it lands, and if we ever want full feature support on wayland it may require a completely different approach if upstream decides they have different priorities, so talking first seems a must.

Smaller issue is that this screencast API previously appeared (to me) to be a freedesktop API (given its vaunted XDG title) with input from at least a couple DEs. But in actuality is just a portion of an ever growing and hairy flatpak control surface (what even is a "GameMode"). At this point the screencast portion of the API is the de-facto API, with compositor specific translators (xdg-desktop-portal-kde and xdg-desktop-portal-wlr) for all major DEs, due to its use by browsers. So while nice that its a single group, it's unfortunately not a collaboration by compositors, who are ultimately the people that need to implement any changes required to support full OBS functionality.

[tytan652]

About PipeWire and Ubuntu 18.04 CIs, it seems that Bionic doesn't provide PipeWire.

[Girgias]

Is this in a usable/testable state, or still completely W.I.P.?
I can trial this on Fedora 33 stock.

[GeorgesStavracas]

Is this in a usable/testable state, or still completely W.I.P.?

This is very much WIP, but it should work decently.

[Girgias]

Is this in a usable/testable state, or still completely W.I.P.?

This is very much WIP, but it should work decently.

Well I'll compile and try it out then. :-)

[Haxk20]

I have tried using it but it seems that it is completely ignoring the FPS settings and runs at what seems like 24fps recording which is well quite slow :) Other than that it works perfectly fine.

[GeorgesStavracas]

I have tried using it but it seems that it is completely ignoring the FPS settings and runs at what seems like 24fps recording which is well quite slow :) Other than that it works perfectly fine.

With this last push, the new capture hints PipeWire about the ideal frame rate, which is what OBS Studio is configured to. The cool thing is that the compositor (GNOME Shell, KWin, etc) also only sends frames at that rate, so e.g. setting 30 FPS should make it lighter on the compositor side.

[Haxk20]

I have tried using it but it seems that it is completely ignoring the FPS settings and runs at what seems like 24fps recording which is well quite slow :) Other than that it works perfectly fine.

With this last push, the new capture hints PipeWire about the ideal frame rate, which is what OBS Studio is configured to. The cool thing is that the compositor (GNOME Shell, KWin, etc) also only sends frames at that rate, so e.g. setting 30 FPS should make it lighter on the compositor side.

Can confirm this works.

Even values above 60Hz work. (Tested on 144Hz monitor recording at 90FPS.)
Great job and ty for fixing the issue.

[YaLTeR]

Just tested this branch, seems to work well after the last update indeed.

Some bikeshedding: perhaps instead of Screen Capture (PipeWire) it should be called Screen Capture (Desktop Portal)? Since really the point is that we're using the portal, and PipeWire just happens to be the transport mechanism.

[tytan652]

Some bikeshedding: perhaps instead of Screen Capture (PipeWire) it should be called Screen Capture (Desktop Portal)? Since really the point is that we're using the portal, and PipeWire just happens to be the transport mechanism.

@YaLTeR
This feature is not Flatpak only then "Desktop Portal" naming can't be used.

[Conan-Kudo]

Desktop Portals are not exclusive to Flatpaks. They are required for Flatpaks, but they are usable without Flatpak.

[tytan652]

Desktop Portals are not exclusive to Flatpaks. They are required for Flatpaks, but they are usable without Flatpak.

Oh, didn't know that

[Conan-Kudo]

That said, in this case, this looks like we're using PipeWire directly, so I'm not sure it's appropriate to generically call it "Desktop Portal"?

[YaLTeR]

That said, in this case, this looks like we're using PipeWire directly, so I'm not sure it's appropriate to generically call it "Desktop Portal"?

The plugin contacts the screencast portal, which then establishes a PipeWire stream between the compositor and the plugin.

[YaLTeR]

The issue of OBS showing the cursor even when a game hides it is still present, so I tried to debug it a little to understand whether Mutter is at fault here or the plugin. I added a cursor valid log right after this line.

In the GTK4 demo in this dialog all 4 squares hide the cursor (in OBS too).

Two of them keep the cursor valid, other two result in cursor valid = false, but all 4 hide the cursor fine in OBS.

In SDL2 there's a relative mouse mode test (test/testrelative) that triggers the issue (cursor hidden on screen but not in OBS) both in X11 and Wayland mode. The cursor always stays valid according to the log.

As far as I can tell in Wayland mode it calls the following code:

        wl_pointer_set_cursor (pointer,
                               input->pointer_enter_serial,
                               NULL,
                               0,
                               0);

This is as far as I got for now, the next step is to check Mutter's handling. It's odd that the GTK4 demo works fine in at least two different ways, yet the SDL2 test breaks it.

[myownfriend]

This doesn't make sense to me since you're not locked to a Pipewire version based on your OS version. For example, I'm on Ubuntu 20.10 so this change wouldn't effect me as I should have Pipewire 0.3.10-4, however I manually upgrade Pipewire so I'm really on the latest version (0.3.24). This should test the version of Pipewire that's installed instead.

Btw, this is my first time responding to specific code on Github so if I did something dumb then I apologize lol

[tytan652]

This doesn't make sense to me since you're not locked to a Pipewire version based on your OS version.

On "Fixed Release" (Debian, Ubuntu, Linux Mint), many packages have their version frozen and only receive security and bug fixes. It means that you are locked to the version supplied by the distribution.

Manually upgrading a dependency package can provoke a breakage in your system because some package were compiled against the official dependency package and can't work with a newer one.

On "Rolling Release", if a dependency package is updated every package which depends on it maybe needs to be rebuilt against the new version.

Pipewire is a dependency for some package, manually upgrading it is at your risk.

An average user doesn't do this, it only uses official repos and maybe some third-party repos for some software like obs-studio.
And obs-studio package is built only against distribution packages.

[GeorgesStavracas]

The Flatpak workflow failure is fixed by #4385

[kkartaltepe]

crash on exit may have been related to some version of pipewire/gnome. I had the old build of xdg i was using and couldnt replicate it today. I also couldnt replicate the crash on the PR's code.

There are some memory leaks, probably dstr changes only freeing the array data and not the dstrs.

Trying to change fps during capture appears to have broken all but one of the captures (with 3 window captures and 2 screen captures, only 1 window capture survived). It logs 5 times 'FPS Changed, renegotiating' but no errors.

Not an issue for this PR, but I think there are some jitter issues (way less jitter than the old build of xdg-obs-portal I compared to). Maybe because pipewire isnt feeding us at exactly 60fps so frames get lost since we have no buffering?

[GeorgesStavracas]

There are some memory leaks, probably dstr changes only freeing the array data and not the dstrs.

Hm, all dstrs I've added are stack allocated, and their arrays are properly freed. There are 2~3 "leaks" reported by OBS on exit, but I've tracked them down to internal global GDBus stuff. I'm not sure how to teach the allocator about these false positives.

Trying to change fps during capture appears to have broken all but one of the captures (with 3 window captures and 2 screen captures, only 1 window capture survived). It logs 5 times 'FPS Changed, renegotiating' but no errors.

On it, but so far it all points to a regression in PipeWire.

[xmixahlx]

this is working fantastic on a variety of debian unstable/experimental/git packages for mesa+pipewire+sway and amdgpu/zen3. i've been using wlrobs prior to this pipewire patch, and your wayland patch prior to landing in master. this pipewire approach does drop resources utilized (~2% to 1%) while feeling smoother and more reliable in usage.

so far the only issue has been the preview being borked after minimizing to tray and unminimizing: you see through to the desktop in the OBS preview window. However, disabling and enabling preview fixes this consistently.

@GeorgesStavracas and everyone else contributing: thank you for your efforts!

[kkartaltepe]

Everything looks good on my machine on the latest commits.

[GeorgesStavracas]

@xmixahlx would you please file a bug report here and tag me there? So I don't forget to take a look at it when possible 🙂

@kkartaltepe I've cherry-picked your commit changing the signature of gs_texture_create_from_dmabuf, and took the liberty to run clang-format on your commit (and squash the second commit into the PipeWire one) - let me know if you have any objection :)

[GeorgesStavracas]

Ah, let's ping @w23 and @nowrep too since this can impact obs-kmsgrab and obs-vkcapture respectively. This is probably the last API change before a stable release.

[kkartaltepe]

I restarted the azure pipelines, probably some weird cloud flakiness.

[nowrep]

Looks fine, the only thing that changed API-wise is added drm format, which is a good thing 👍