Can't upgrade to matrix-synapse-py3 1.19.2 on Ubuntu bionic

[kalon33]

On Ubuntu Bionic, it is not possible to upgrade to latest matrix-synapse-py3 as synapse won't start due to missing dependencies: Missing Requirements: 'attrs>=19.1.0', 'typing-extensions>=3.7.4'.

Please fix.

Sep 17 18:07:03 spiralo-new2 systemd[1]: Starting Synapse Matrix homeserver...
-- Subject: Unit matrix-synapse.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit matrix-synapse.service has begun starting up.
Sep 17 18:07:04 spiralo-new2 matrix-synapse[25830]: ERROR:root:Needed attrs>=19.1.0, got attrs==18.2.0
Sep 17 18:07:04 spiralo-new2 matrix-synapse[25830]: ERROR:root:Needed typing-extensions>=3.7.4, got typing-extensions==3.6.6
Sep 17 18:07:04 spiralo-new2 matrix-synapse[25830]: Missing Requirements: 'attrs>=19.1.0', 'typing-extensions>=3.7.4'
Sep 17 18:07:04 spiralo-new2 matrix-synapse[25830]: To install run:
Sep 17 18:07:04 spiralo-new2 matrix-synapse[25830]:     pip install --upgrade --force 'attrs>=19.1.0' 'typing-extensions>=3.7.4'
Sep 17 18:07:04 spiralo-new2 systemd[1]: matrix-synapse.service: Control process exited, code=exited status=1
Sep 17 18:07:04 spiralo-new2 systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
Sep 17 18:07:04 spiralo-new2 systemd[1]: Failed to start Synapse Matrix homeserver.
-- Subject: Unit matrix-synapse.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit matrix-synapse.service has failed.
-- 
-- The result is RESULT.

[kalon33]

I tried to perform pip command into the venv but updated python packages are not taken into account.

[clokep]

Note that #7681 bumped the attrs version.

How have you installed Synapse? Is this using packages?

[kalon33]

I upgraded Synapse from an older version, using the official package named matrix-synapse-py3 (I added the repo in my repo list, but the corresponding deb package is here: https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.19.2+bionic1_amd64.deb )

[clokep]

Thanks! Do you know what the older version was? (Was it 1.19.1 or something older than that?)

[kalon33]

I upgraded from 1.19.1 and never had this problem before.

[kalon33]

it seems that python dependencies provided for the virtualenv weren't updated in the package

[erikjohnston]

This is weird, the attrs library was bumped way back in v1.16.0, and there should be no changes in the dependencies between v1.19.1 and v1.19.2. Is it possible that somehow synapse is using the wrong virtual env?

[kalon33]

@erikjohnston the venv is in /opt/venvs/matrix-synapse/ and seems up-to-date now, but it seems that it uses system python and its deps, and not the one from the venv... I removed attrs and typing-extensions from the system, and now I got theses errors:

-- Unit matrix-synapse.service has begun starting up.
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]: Traceback (most recent call last):
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     "__main__", mod_spec)
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     exec(code, run_globals)
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/app/homeserver.py", line 31, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet import defer, reactor
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/reactor.py", line 38, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet import default
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/default.py", line 56, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     install = _getInstallFunction(platform)
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/default.py", line 44, in _getInstallFunction
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet.epollreactor import install
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/epollreactor.py", line 24, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet import posixbase
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 18, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet import error, udp, tcp
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/udp.py", line 53, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet import base, defer, address
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/base.py", line 27, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet._resolver import (
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/_resolver.py", line 25, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     from twisted.internet.address import IPv4Address, IPv6Address
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:   File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/address.py", line 101, in <module>
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]:     @attr.s(hash=False, repr=False, eq=False)
Sep 20 18:02:47 spiralo-new2 matrix-synapse[19132]: TypeError: attrs() got an unexpected keyword argument 'eq'
Sep 20 18:02:47 spiralo-new2 systemd[1]: matrix-synapse.service: Control process exited, code=exited status=1
Sep 20 18:02:47 spiralo-new2 systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
Sep 20 18:02:47 spiralo-new2 systemd[1]: Failed to start Synapse Matrix homeserver.

[erikjohnston]

I can't really think of a reason why Synapse would prefer the libraries from the system rather than the venv, in fact that's the entire point of a venv 😕

That particular error looks like Twisted wants a more up to date attrs, so you may just need to update it in the venv? It might be worth completely uninstalling attrs in the venv and then reinstalling it?

[richvdh]

That particular error looks like Twisted wants a more up to date attrs, so you may just need to update it in the venv? It might be worth completely uninstalling attrs in the venv and then reinstalling it?

I wouldn't recommend messing about with the stuff in the venv that is installed by the package. Better to reinstall the package than start removing things from the venv, otherwise you're going to get into a mess.

In terms of the root problem: have you modified the systemd service file that is used tostart synapse? To confirm, please can you share the output of sudo systemd cat matrix-synapse.service ?

[kalon33]

@richvdh Hi, I don't remember having edited it, here is the matrix-synapse.service file (your command line wasn't working, so I too the liberty to modify it):

$ sudo cat /etc/systemd/system/multi-user.target.wants/matrix-synapse.service 
[Unit]
Description=Synapse Matrix homeserver

[Service]
Type=simple
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=/etc/default/matrix-synapse
ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
SyslogIdentifier=matrix-synapse

[Install]
WantedBy=multi-user.target

[richvdh]

ok, that looks fine. (Sorry, I meant systemctl rather than systemd, but what you did is fine)

I'm not seeing anything in the output above that suggests that it's using either the system python, or preferring the system libraries. It's expected that runpy comes from /usr/lib/.

Can you share the output of the following:

/opt/venvs/matrix-synapse/bin/python -c "import attr; print(attr.__file__); print(attr.__version__)"

[richvdh]

I've removed the "release blocker" label as this seems to be an isolated incident rather than a problem with the package.

[kalon33]

ok, that looks fine. (Sorry, I meant systemctl rather than systemd, but what you did is fine)

I'm not seeing anything in the output above that suggests that it's using either the system python, or preferring the system libraries. It's expected that runpy comes from /usr/lib/.

Can you share the output of the following:

/opt/venvs/matrix-synapse/bin/python -c "import attr; print(attr.__file__); print(attr.__version__)"

$ /opt/venvs/matrix-synapse/bin/python -c "import attr; print(attr.__file__); print(attr.__version__)"
/usr/lib/python3.6/attr/__init__.py
18.2.0

So it seems to use the system attr for an unknown reason...

[richvdh]

ok, the problem is that things are installed in /usr/lib/python3.6. That should only be used for system python libraries (of which attr is not one), and does indeed take precedence over the virtualenv:

rav@ubuntu18-04:~$ /opt/venvs/matrix-synapse/bin/python -c "import sys; print(sys.path)"
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/opt/venvs/matrix-synapse/lib/python3.6/site-packages']

so the question becomes: how did you manage to install an attr package there, and can you undo it?

[kalon33]

I don't know, pip (python3.6 -m pip) doesn't know about it. I removed it "manually", checked that I didn't have a remaining typing-extension and now it seems to work properly. Thanks for your help!