Crashes on startup while generating presence stream

[matrixbot]

This issue has been migrated from #10106.

---

Description

Crashes on startup while generating presence stream

Steps to reproduce

• HS is running a set of workers: event-persister, federation-{inbound,reader,sender}, presence-writer
• All workers and main process were restarted
• event-persister and federation-sender crashed 3 times each on startup before successfully starting

Event Persister:

2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,318 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,319 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,321 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,327 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,328 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,330 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,331 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,343 - twisted - 258 - ERROR - sentinel-     return _run_code(code, main_globals, None,
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel-     exec(code, run_globals)
2021-06-02 07:46:12,344 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 483, in <module>
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel-     start(sys.argv[1:])
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel-     hs.setup()
2021-06-02 07:46:12,345 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel-     self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel-     main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,346 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,347 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel-     self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,348 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel-     self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,349 - twisted - 258 - ERROR - sentinel-     cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel-     self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,350 - twisted - 258 - ERROR - sentinel-     return func(sql, *args)
2021-06-02 07:46:12,351 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

Federation Sender:

2021-06-02 07:46:12,819 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,820 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for public_room_list_stream(stream_id)
2021-06-02 07:46:12,821 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for push_rules_stream(stream_id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for pushers(id)
2021-06-02 07:46:12,826 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for deleted_pushers(stream_id)
2021-06-02 07:46:12,827 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for local_group_updates(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_stream(stream_id)
2021-06-02 07:46:12,829 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for user_signature_stream(stream_id)
2021-06-02 07:46:12,830 - synapse.storage.util.id_generators - 55 - INFO - main- initialising stream generator for device_lists_outbound_pokes(stream_id)
2021-06-02 07:46:12,841 - twisted - 258 - ERROR - sentinel- Traceback (most recent call last):
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel-     return _run_code(code, main_globals, None,
2021-06-02 07:46:12,842 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel-     exec(code, run_globals)
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/app/federation_sender.py", line 23, in <module>
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel-     start(sys.argv[1:])
2021-06-02 07:46:12,843 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py", line 470, in start
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel-     hs.setup()
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/server.py", line 280, in setup
2021-06-02 07:46:12,844 - twisted - 258 - ERROR - sentinel-     self.datastores = Databases(self.DATASTORE_CLASS, self)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 76, in __init__
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel-     main = main_store_class(database, db_conn, hs)
2021-06-02 07:46:12,845 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 553, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/user_directory.py", line 40, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/stats.py", line 103, in __init__
2021-06-02 07:46:12,846 - twisted - 258 - ERROR - sentinel-     super().__init__(database, db_conn, hs)
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/presence.py", line 46, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel-     self._presence_id_gen = MultiWriterIdGenerator(
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 283, in __init__
2021-06-02 07:46:12,847 - twisted - 258 - ERROR - sentinel-     self._load_current_ids(db_conn, tables)
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 304, in _load_current_ids
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel-     cur.execute(sql, (self._stream_name, self._writers))
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 290, in execute
2021-06-02 07:46:12,848 - twisted - 258 - ERROR - sentinel-     self._do_execute(self.txn.execute, sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel-   File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 316, in _do_execute
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel-     return func(sql, *args)
2021-06-02 07:46:12,849 - twisted - 258 - ERROR - sentinel- psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

Raising as this HS is testing out the presence worker and the presence stream appears to be the problematic one.

Version information

• Homeserver: vector.modular.im
• Version: 1.34.0