faster server startup

[totaam]

Following the complaint that xpra startup is slow (ie: in this ticket: mviereck/x11docker#167), we can parallelize some of the mixin startup work.

Related to #1838

[totaam]

Updates:

• r23025: if opengl is set to noprobe, just skip opengl_probe() - saves almost a second
• r23026: no need to populate the platform cache before registering the SIGCHLD handler (saves about 100 milliseconds)
• r23027: move some webcam code to the initialization thread (saves around 50 milliseconds)
• r23028: change fake-xinerama option so we can find the library path once at build time (saves about 100 milliseconds)

[totaam]

Remaining places where we could save time:

• instantiating the server class (loading bindings: can we delay that?): 70ms
• X11ServerBase.init(opts): 90ms
• init_randr: 10ms (could use --resize-display=no to skip?)
• mdns_publish: 20ms (could start from init thread?)
• init_encodings: 70ms (loading all the bindings: only do this from the init thread?)
• idle_add takes 90ms the first time we call it (just the main loop starting?)

[totaam]

• instantiating the server class is slow because we load so many modules (lots of mixins) and cython extensions (x11 bindings mostly), but there is one place where we can do better: importing xpra.net.net_util takes too long: over 40ms!
• X11ServerBase.init(opts) is slow because of GTK's selection.acquire() in XSettingsManager - not much we can do about that
• print_run_info takes a bit of time on Posix because we lookup the user in the password database: move this to a new thread or use the background worker (no need to delay the init thread for this)

[totaam]

2019-06-26 17:58:18: antoine uploaded file server-start-debug.patch (29.4 KiB)

ugly debug logging patch

[totaam]

Updates:

• r23034 (missed from r23028)
• r23030 print_run_info via worker thread
• r23032 use faster socket functions for if_nametoindex and if_indextoname with python3
• r23037 call mdns_publish from the worker thread
• r23038 load codecs from threaded setup
• r23039 don't watch for menu changes when we don't expose them (start-new-commands=no)

With these changes and the following command line:

python3 /usr/bin/xpra start --no-daemon --start=xterm \
   --opengl=noprobe --xsettings=no --no-speaker --no-microphone --html=no

The startup takes at most 350ms on my laptop. (400ms for python2)

Other places where we still spend a little bit too much time:

• server base import of mixins: ~20ms (nothing we can do)
• BaseHTTPRequestHandler takes 30ms to import! (can be skipped with --html=off - nothing we can do)
• main loop startup: over 100ms (not sure what can be done there)

New potential issue: race condition if we are still initializing via the worker thread when we are asked to shutdown.. maybe process the shutdown request via the worker thread to guarantee we complete what has already been queued?

[totaam]

New issues found in https://github.com/mviereck/x11docker/files/3341648/xpraserver.log:

• the server is started against an existing vfb so we end up trying to locate an existing _XPRA_UINPUT_ID and this takes a while (0.4s):

2019-06-29 10:10:12,666 Popen(['xprop', '-display', ':101', '-root', '_XPRA_UINPUT_ID'])=\
    (0, '_XPRA_UINPUT_ID:  no such atom on any window.\n', '')
2019-06-29 10:10:13,038 setup_local_sockets: bind=['auto']

Partially dealt with in r23043. I'm not sure we want to spend more time on this particular issue: re-using an existing display is not a standard setup..

• maybe add an option (env var) to skip the codec self tests? (30ms) meh - now that we do this in the init thread, it doesn't really matter much

[totaam]

See also #2344 and #2347

[totaam]

2019-06-30 11:22:25: antoine uploaded file server-start-debug-mainloop.patch (4.4 KiB)

shows that the main loop startup is not slow because of anything we do

[totaam]

Updates:

• r23047 fix race condition with worker thread
• r23048 fix race condition with initialization thread
• r23049 better worker thread cleanup: faster, safer
• as per the patch above, the main loop takes 150ms to start, but this is not caused by our code

With all these changes, a server starts on my laptop in under 4 seconds flat - and that's including the slow xvfb startup!
With an existing display, the total startup time (logged) is now around 250ms! (actual time is marginally higher than that since logging starts after a the initial startup code)

Log sample with an existing display:

$ python3 /usr/bin/xpra start --no-daemon --start=xterm \
    --opengl=noprobe --xsettings=no --speaker=no --microphone=no \
    --mdns=no --start-new-commands=no --notifications=no --html=no \
    :10 --use-display
2019-06-30 11:15:27,722 created unix domain socket: /run/user/1000/xpra/linux.home-10
2019-06-30 11:15:27,766 pointer device emulation using XTest
2019-06-30 11:15:27,787 xvfb pid=20443
2019-06-30 11:15:27,959 xpra is ready.
2019-06-30 11:15:27,960 xpra GTK3 X11 version 3.0-[r22755](../commit/8312a85c736df6c985daba8f42c9d796e3fc7873) 64-bit
2019-06-30 11:15:28,037  uid=1000 (antoine), gid=1000 (antoine)
2019-06-30 11:15:28,038  running with pid 20892 on Linux Fedora 30 Thirty
2019-06-30 11:15:28,038  connected to X11 display :10 with 24 bit colors
2019-06-30 11:15:28,657 found 1 virtual video device for webcam forwarding
2019-06-30 11:15:28,750 7.7GB of system memory

[totaam]

r23066: don't load xdg menu data unless start-new-commands is enabled - huge savings

[totaam]

See also:

• faster client startup #2347 client
• cache gstreamer config #2349 cache audio query

[totaam]

This may have caused a regression: #2355.

And definitely caused this bug: #2180

[totaam]

New ticket for 4.1: #2815