Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Commit

Permalink
devices: use combined ANY clause for faster cleanup
Browse files Browse the repository at this point in the history
Old device entries for the same user were being removed in individual
SQL commands, making the batch take way longer than necessary.

This combines the commands into a single one with a IN/ANY clause.

Example of log entry before the change, regularly observed with
"log_min_duration_statement = 10000" in PostgreSQL's config:

    LOG:  duration: 42538.282 ms  statement:
    DELETE FROM device_lists_stream
    WHERE user_id = '@Someone' AND device_id = 'someid1'
    AND stream_id < 123456789
    ;
    DELETE FROM device_lists_stream
    WHERE user_id = '@Someone' AND device_id = 'someid2'
    AND stream_id < 123456789
    ;
    [repeated for each device ID of that user, potentially a lot...]

With the patch applied on my instance for the past couple of days, I
no longer notice overly long statements of that particular kind.
  • Loading branch information
pacien committed Jul 2, 2023
1 parent 13fc891 commit 933daec
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions synapse/storage/databases/main/devices.py
Original file line number Diff line number Diff line change
Expand Up @@ -1950,12 +1950,16 @@ def _add_device_change_to_stream_txn(

# Delete older entries in the table, as we really only care about
# when the latest change happened.
txn.execute_batch(
"""
cleanup_obsolete_stmt = """
DELETE FROM device_lists_stream
WHERE user_id = ? AND device_id = ? AND stream_id < ?
""",
[(user_id, device_id, min_stream_id) for device_id in device_ids],
WHERE user_id = ? AND stream_id < ? AND %s
"""
device_ids_clause, device_ids_args = make_in_list_sql_clause(
txn.database_engine, "device_id", device_ids
)
txn.execute(
cleanup_obsolete_stmt % (device_ids_clause,),
[user_id, min_stream_id] + device_ids_args,
)

self.db_pool.simple_insert_many_txn(
Expand Down

0 comments on commit 933daec

Please sign in to comment.