py3k server support

[totaam]

Issue migrated from trac ticket # 853

component: platforms | priority: minor | resolution: fixed

2015-05-08 06:18:06: antoine created the issue

---

Python3 tracker ticket: #1568.

Eventually, we will have to convert the server code to py3k... Which is going to be a pain, probably even worse than #90.

We should probably move away from GTK at that point too, and replace all the pygobject plumbing with something a bit simpler and faster.

This ticket is here to record all the things that will need to be done to make this happen.

Apart from the usual string vs unicode nonsense, we will have to deal with the borken change they made to dict items vs iteritems in version 3: we rely on the fact that items() makes a copy, which is no longer true, we can probably switch to viewkeys() for most cases - except that this is python 2.7 only... so we'll need to drop support for 2.6 before that (ie: all centos 6).

[totaam]

2017-09-17 17:14:37: antoine changed status from new to assigned

[totaam]

2017-09-17 17:14:37: antoine edited the issue description

[totaam]

2017-11-13 05:52:18: antoine commented

---

Socket fixes for running the proxy server with python3: r17410.
It still fails shortly after:

Exception in thread new-tcp-connection:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.6/site-packages/xpra/server/server_core.py", line 861, in handle_new_connection
    sock.settimeout(self._socket_timeout)
OSError: [Errno 9] Bad file descriptor

Very relevant discussion: How to create a socket.socket() object from a socket fd?.
Python3 does this:

def fromfd(fd, family, type, proto=0):
    """ fromfd(fd, family, type[, proto]) -> socket object

    Create a socket object from a duplicate of the given file
    descriptor.  The remaining arguments are the same as for socket().
    """
    nfd = dup(fd)
    return socket(family, type, proto, nfd)

But do we want this fd to be dup-ed here?

[totaam]

2018-10-22 15:19:52: antoine commented

---

Huge changes:

• r20756: preparatory work: move wm_check, define compat functions
• r20757: clipboard, gdk window, more compat functions
• r20758: usual strings vs bytes python3 nonsense
• r20759: window models
• r20760: world window
• r20761: composite helper, fix models
• r20762: desktop server

The desktop server (start-desktop) now runs OK, the seamless server still needs work (crashes hard).

[totaam]

2018-10-23 08:34:47: antoine commented

---

• r20770: fix displayfd
• r20772: simplify code (lookup xid just once)
• r20775: ensure we only initialize the display once
• r20776: import cleanup
• r20777: gdk display init refactoring, strings vs bytes, dict iterators, disable world window with GTK3
• r20781 minor refactoring cleanup

Related:

• r20771: add missing call to gobject initializer (explains why we had to set all the values)
• move away from GTK: replace GTK glue wherever possible #1995
• r20773: duplicated header import

TODO:

• decide what to do with the world window: fix it? don't use it? (risk breaking compatibility with some Java apps?) - see also xterm regression: popup window does not get focus #1999
• the desktop manager is also missing some flags that have been remove in GTK3 (gtk.NO_WINDOW) - still works!?
• display closing issues: SIGINT causes crash when not already using --use-display=yes
• re-check all server types
• platform checks

[totaam]

2018-10-25 10:55:15: antoine commented

---

Updates:

• r20794: win32 and macos platform fix
• r20795, r20811, r20824: strings vs bytes nonsense
• r20797: some renaming missed from earlier refactoring
• r20822: use python object attributes instead of "data" functions that no longer exist, r20823: same but replace weak key dict
• r20825: make window initialization code more resilient
• 20810: nvenc python2 vs python3 vs cython version failures

Still crashes hard when we close any window - simplified gdb backtrace:

#5  0x00007fffee190f83 in gdk_x_error (xdisplay=0x555555ab8030, error=0x7fffffff9c40) at gdkmain-x11.c:307
#11 0x00007fffee154da4 in gdk_flush () at gdkdisplay.c:598
#60 0x00007fffee18b39f in gdk_event_apply_filters (xevent=xevent@entry=0x7fffffffba70, event=event@entry=0x5555567ec200, window=window@entry=0x0) at gdkeventsource.c:79
#65 0x00007fffee7b726d in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#68 0x00007fffed4f3b5d in gtk_main () at gtkmain.c:1323

Something is not right with our gdk event filter - same code runs fine with GTK2..

[totaam]

2018-10-25 15:42:25: antoine commented

---

More updates:

• r20832: fix clipboard error (strings vs bytes again)
• r20833: fix server crash when closing a window
• r20834: strings vs bytes (desktop names)
• r20835 + r20836: fix key mapping for non-native clients (ie: win32)

[totaam]

2018-10-27 06:34:28: antoine commented

---

Updates:

• r20837: minor compatibility fix
• r20839: fix bencode packet encoder - also means we drop support for ancient bencode versions, and get a performance boost
• r20841: enable most x11 server unit tests with python3
• r20842 + r20843: fix yaml packet encoder, better protocol level error message

Some documentation that may be helpful:

• GdkX11.X11Window
• GdkX11.functions
  Not sure we want to use those functions as this will tie us with Gdk more, not less.

[totaam]

2018-10-31 11:07:01: mjharkin commented

---

I think r20770 broke python2 functionaility.
1st login gives a server error:

2018-10-31 11:50:51,843 start_server_subprocess failed
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 289, in proxy_session
    proc, socket_path, display = self.start_new_session(username, uid, gid, sns, displays)
  File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 462, in start_new_session
    proc, socket_path, display = start_server_subprocess(sys.argv[0], args, mode, opts, username, uid, gid, env, cwd)


  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1768, in start_server_subprocess
    buf = read_displayfd(r_pipe, proc=None) #proc deamonizes!
  File "/usr/lib64/python2.7/site-packages/xpra/platform/displayfd.py", line 49, in read_displayfd
    if e.errno!=errno.EINTR:
AttributeError: 'error' object has no attribute 'errno'
2018-10-31 11:50:51,844 Error: failed to start server subprocess:
2018-10-31 11:50:51,845  'error' object has no attribute 'errno'
2018-10-31 11:50:51,845 disconnect(server error, ('failed to start a new session',))

2nd login is successful.

[totaam]

2018-10-31 12:48:34: antoine commented

---

@mjharkin: thanks for the report, r20887 should fix this.

[totaam]

2018-10-31 13:38:35: mjharkin commented

---

Yes, fixed in r20887. Thanks.

[totaam]

2019-02-19 16:27:58: antoine commented

---

r21729 fixes the keyboard setup code - probably more fixes needed here to support keycode translation (stoopid bytes vs strings in python3)

[totaam]

2019-02-20 04:15:53: antoine commented

---

• r21721 dbus --use-display bug fix
• r21715 implement calc_constrained_size for python3

[totaam]

2019-02-20 05:28:05: antoine commented

---

• r21737 fixes the websocket handler (python3 http and websocket handler fails #2149)

[totaam]

2019-02-21 05:07:24: antoine commented

---

r21777 finally fixes the video encoders (more py3k string nonsense), mostly just by undoing r20658.

[totaam]

2019-02-22 13:00:14: antoine changed status from assigned to closed

[totaam]

2019-02-22 13:00:14: antoine set resolution to fixed

[totaam]

2019-02-22 13:00:14: antoine commented

---

This is ready for release.
It should be on par with the python2 server.

[totaam]

2019-02-28 11:14:07: antoine commented

---

Regression with dbus: #2181

[totaam]

2019-03-13 12:57:03: antoine commented

---

More fixes: r22073, r22074, r22075. (and also more issues: #2201, #2209, #2211)