Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

consider handling incoming replication queues in a more efficient way #7444

Closed
richvdh opened this issue May 6, 2020 · 2 comments · Fixed by #7861
Closed

consider handling incoming replication queues in a more efficient way #7444

richvdh opened this issue May 6, 2020 · 2 comments · Fixed by #7861
Assignees
@richvdh
Labels
A-Performance Performance, both client-facing and admin-facing A-Workers Problems related to running Synapse in Worker Mode (or replication)

Comments

@richvdh
Copy link
Member

richvdh commented May 6, 2020

if we get behind on replication (eg because the connection dropped and we have to process a POSITION, which is slowish), we stack up background processes on a linearizer; bg processes are heavy (particularly with respect to prometheus metrics) and linearizers aren't terribly efficient once the queue gets long either.
@neilisfragile neilisfragile added the A-Performance Performance, both client-facing and admin-facing label May 14, 2020
@richvdh richvdh mentioned this issue Jul 1, 2020
Open
@richvdh
Copy link
Member Author

richvdh commented Jul 1, 2020

I would like to see this happening asap as a (partial) fix to #7744.

@richvdh
Copy link
Member Author

richvdh commented Jul 1, 2020

I believe we could implement this more efficiently by maintaining a list of replication updates to work through and having a single "thread" working through the list. This is a pattern we use in various other places in the codebase (search for is_processing for examples).

richvdh added a commit that referenced this issue Jul 15, 2020
@richvdh
Optimise queueing of inbound replication commands
1bc4e93 
When we get behind on replication, we tend to stack up background processes
behind a linearizer. Bg processes are heavy (particularly with respect to
prometheus metrics) and linearizers aren't terribly efficient once the queue
gets long either.

A better approach is to maintain a queue of requests to be processed, and
nominate a single process to work its way through the queue.

Fixes: #7444
@richvdh richvdh mentioned this issue Jul 15, 2020
Merged
@richvdh richvdh self-assigned this Jul 15, 2020
erikjohnston pushed a commit that referenced this issue Jul 16, 2020
@richvdh
Optimise queueing of inbound replication commands (#7861)
e530006 
When we get behind on replication, we tend to stack up background processes
behind a linearizer. Bg processes are heavy (particularly with respect to
prometheus metrics) and linearizers aren't terribly efficient once the queue
gets long either.

A better approach is to maintain a queue of requests to be processed, and
nominate a single process to work its way through the queue.

Fixes: #7444
@richvdh richvdh added the A-Workers Problems related to running Synapse in Worker Mode (or replication) label Feb 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@richvdh richvdh

Labels
A-Performance Performance, both client-facing and admin-facing A-Workers Problems related to running Synapse in Worker Mode (or replication)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optimise queueing of inbound replication commands matrix-org/synapse
2 participants
@neilisfragile @richvdh