Performance issues with GLX backend and 3D windows

[bucaneer]

Background: I've been experimenting with various vsync options trying to find the setup that performs best. I couldn't achieve truly tear-free display with --backend xrender, and the vsync implementation in nvidia proprietary drivers seems to be rather slow. After some tweaking, I arrived at this setup:

compton -cCzG -t-3 -l-5 -r4 \
 --config /dev/null --backend glx  --dbe \
 --vsync opengl-swc --sw-opti --refresh-rate 60 \
 --shadow-exclude "name = 'synapse'" \
 --shadow-ignore-shaped --unredir-if-possible \
 --glx-no-stencil --glx-copy-from-front --paint-on-overlay \
 --glx-swap-method copy --glx-no-rebind-pixmap

Most of the time, it works perfectly: no tearing, no latency, virtually no CPU use.

Problem: However, this setup doesn't like windows with 3D content. That would be mostly games, but for testing purposes glxgears is enough to reproduce all symptoms. Basically, if I move the 3D window around, or move any other window so that it overlaps the 3D window, compton's CPU usage skyrockets and the whole display may freeze for several seconds at a time.

Curiously, this behaviour seems to be related to the priority (nice level) of the 3D app - the lower priority, the better. For example, nice -n10 glxgears no longer causes freezing. But the priority of compton process has very little influence - perhaps the framerate decreases at lower priority, but it is not possible to cause or prevent freezing by setting compton's nice level. However, since I'm mostly concerned about running games in windowed mode, reduced priority may lead to reduced performance, which I'd like to avoid.

Is there some launch option I overlooked, or some other tweak I could use that could fix the issue?

Additional info:
Compton version: git-v0.1_beta2-13-g6772ffb-2014-01-08 (same behaviour with the version form 2013-11-10 that I used before today)
GPU: Nvidia GTX 650 Ti, driver version 331.20
CPU: Core2Duo E8400, 3.0GHz
Window manager: Openbox (3.5.2)

[richardgv]

1. Do not use --glx-copy-from-front. It was intended as an optimization but in fact it generally greatly slows compton down because glCopyPixels() is slow as hell with most drivers.
2. Do not use --sw-opti with --vsync. It could lead to reduced frame rate.
3. As you may have noticed --dbe does nothing under GLX backend.
4. --glx-swap-method copy is generally broken on nvidia-drivers (unless you have "Allow Flipping" disabled in nvidia-settings -- which should definitely be enabled), and typically only --glx-swap-method 3 is safe. How did you get it working without artifacts?
5. It could be slow to drag a OpenGL window, but it probably isn't a performance issue: Basically it appears really sluggish when I drag a VSync-ed GL window in a VSync-ed compton, and once I disable "Sync to VBlank" in nvidia-settings (but you could keep --vsync opengl-swc in compton, so the app isn't VSync-ed while compton is) the issue goes away.
6. Don't use glxgears for testing this. It's intended to exhaust all your resources by painting as fast as possible and generally an OpenGL app will not do that. I use supertux-0.3.4 as a more realistic testing target.
7. Dragging windows will bring very high CPU load, but if it occurs when you are (almost) doing nothing, a system-wide profiling may show something valuable.
8. We have --backend xr_glx_hybrid as another possible choice, by the way.
9. Your discovery about niceness is interesting! :-)

[bucaneer]

Thank you. After your suggestions, my setup now looks like this:

compton -cCzG -t-3 -l-5 -r4 \
 --config /dev/null --backend glx \
 --vsync opengl-swc \
 --shadow-exclude "name = 'synapse'" \
 --shadow-ignore-shaped --unredir-if-possible \
 --glx-no-stencil --paint-on-overlay \
 --glx-swap-method 3 --glx-no-rebind-pixmap

This seems to have fixed the problem in most cases (tested about 10 native and Wine games) - compton's CPU usage stays low and I can move OpenGL windows around about as well as any other kind. It seems that the biggest factor was --glx-swap-method. I had previously disabled flipping because it seemed that copy offered better performance, but now it appears I was mistaken.

However, the issue persists in the games where I originally encountered it - Europa Universalis 4 and Crusader Kings 2, both by the same developer and using the same engine. So it's almost certainly the games' fault - for some reason, they clash with compton. For example, when idling in EU4 main menu without compton running, it steadily uses 42-44% CPU. But if compton is running, EU4 uses 45-62% CPU, and compton uses another 2-35% (with other games, compton always stays at around 2%). If I start moving the game window, compton's CPU usage may spike to 100%, leading to freezes.

So even though it's not really compton's fault, would you have a guess about what in the game engine could be causing this? Or a test to determine that? I'd like to provide a useful bug report to the game developers about this.

Oh, and --backend xr_glx_hybrid seems to perform about as well as glx, except it sometimes produces flickering artifacts. I mostly see them in xfce4-panel - some item in it, like the clock, workspace switcher, CPU temperature monitor, etc, begins rapidly flickering between its proper appearance and a white rectangle for no apparent reason. The flickering either goes away by itself after a few seconds, or when I move mouse pointer over it. I also saw it happened on a newly focused window, but can't seem to reproduce that.

[richardgv]

However, the issue persists in the games where I originally encountered it - Europa Universalis 4 and Crusader Kings 2, both by the same developer and using the same engine. So it's almost certainly the games' fault - for some reason, they clash with compton. For example, when idling in EU4 main menu without compton running, it steadily uses 42-44% CPU. But if compton is running, EU4 uses 45-62% CPU, and compton uses another 2-35% (with other games, compton always stays at around 2%). If I start moving the game window, compton's CPU usage may spike to 100%, leading to freezes.
So even though it's not really compton's fault, would you have a guess about what in the game engine could be causing this? Or a test to determine that? I'd like to provide a useful bug report to the game developers about this.

1. The game might be calling glXSwapIntervalSGI() or some other VSync stuffs that brings the conflict back. If you have a VSync option in the game, turn it off. You could check if it's calling them with apitrace.
2. System-wide profiling might bring some clues if CPU usage is a problem for you.

Oh, and --backend xr_glx_hybrid seems to perform about as well as glx, except it sometimes produces flickering artifacts. I mostly see them in xfce4-panel - some item in it, like the clock, workspace switcher, CPU temperature monitor, etc, begins rapidly flickering between its proper appearance and a white rectangle for no apparent reason. The flickering either goes away by itself after a few seconds, or when I move mouse pointer over it. I also saw it happened on a newly focused window, but can't seem to reproduce that.

1. The fact that xr_glx_hybrid works as bad as glx might imply it's more likely an issue in nvidia-drivers.
2. --vsync-use-glfinish might reduce the flickering, but you don't need try if i isn't appearing better than GLX backend.

[bucaneer]

Huh, after messing with various game settings I think I fixed it. Not fully sure what the exact cause was - refresh rate settings were probably part of it, but I also needed to change some other things I had configured for the game. In any case, now everything is smooth and tear-free. Thanks again for your help.