Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Optimize filter_events_for_client for faster /messages - v2 #14527

Merged

Conversation

MadLittleMods
Copy link
Contributor

@MadLittleMods MadLittleMods commented Nov 22, 2022

Optimize filter_events_for_client -> _get_state_groups_from_groups for faster /messages. Seems like it will make it a magnitude faster (see comment below for actual matrix.org results). As discussed at #14494 (comment) (which was a v1 of this PR)

Fix #14108

Part of #13356 (comment)

SQL query optimization

More context on how we arrived at these queries is available at #14494 (comment)

Before

One query that takes ~80ms: https://explain.depesz.com/s/OtpN

Before raw query and EXPLAIN ANALYZE
EXPLAIN ANALYZE WITH RECURSIVE state(state_group) AS (
    VALUES(739988088::bigint)
    UNION ALL
    SELECT prev_state_group FROM state_group_edges e, state s
    WHERE s.state_group = e.state_group
)
SELECT DISTINCT ON (type, state_key)
    type, state_key, event_id
FROM state_groups_state
WHERE
    state_group IN (
        SELECT state_group FROM state
    )
    AND ((type = 'm.room.history_visibility' AND state_key = '') OR (type = 'm.room.member' AND state_key = '@madlittlemods:matrix.org'))
ORDER BY type, state_key, state_group DESC;
                                                                                          QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=5274.51..5274.52 rows=1 width=87) (actual time=76.555..76.561 rows=2 loops=1)
   CTE state
     ->  Recursive Union  (cost=0.00..284.28 rows=101 width=8) (actual time=0.002..0.443 rows=59 loops=1)
           ->  Result  (cost=0.00..0.01 rows=1 width=8) (actual time=0.001..0.002 rows=1 loops=1)
           ->  Nested Loop  (cost=0.57..28.23 rows=10 width=8) (actual time=0.006..0.007 rows=1 loops=59)
                 ->  WorkTable Scan on state s  (cost=0.00..0.20 rows=10 width=8) (actual time=0.000..0.000 rows=1 loops=59)
                 ->  Index Only Scan using state_group_edges_unique_idx on state_group_edges e  (cost=0.57..2.79 rows=1 width=16) (actual time=0.006..0.006 rows=1 loops=59)
                       Index Cond: (state_group = s.state_group)
                       Heap Fetches: 58
   ->  Sort  (cost=4990.23..4990.23 rows=1 width=87) (actual time=76.554..76.555 rows=2 loops=1)
         Sort Key: state_groups_state.type, state_groups_state.state_key, state_groups_state.state_group DESC
         Sort Method: quicksort  Memory: 25kB
         ->  Nested Loop  (cost=3.11..4990.22 rows=1 width=87) (actual time=0.740..76.542 rows=2 loops=1)
               ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=0.484..0.498 rows=59 loops=1)
                     Group Key: state.state_group
                     Batches: 1  Memory Usage: 24kB
                     ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=0.004..0.459 rows=59 loops=1)
               ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.84..49.37 rows=1 width=87) (actual time=0.009..1.289 rows=0 loops=59)
                     Index Cond: (state_group = state.state_group)
                     Filter: (((type = 'm.room.history_visibility'::text) AND (state_key = ''::text)) OR ((type = 'm.room.member'::text) AND (state_key = '@madlittlemods:matrix.org'::text)))
                     Rows Removed by Filter: 1495
 Planning Time: 3.309 ms
 Execution Time: 76.629 ms
(23 rows)

Time: 80.624 ms

After

One query that takes ~8ms: https://explain.depesz.com/s/Qi4A

After raw query and EXPLAIN ANALYZE
EXPLAIN ANALYZE WITH RECURSIVE sgs(state_group) AS (
    VALUES(739988088::bigint)
    UNION ALL
    SELECT prev_state_group FROM state_group_edges e, sgs s
    WHERE s.state_group = e.state_group
)
(
    SELECT DISTINCT ON (type, state_key)
        state_group, type, state_key, event_id
    FROM state_groups_state
    INNER JOIN sgs USING (state_group)
    WHERE (type = 'm.room.member' AND state_key = '@madlittlemods:matrix.org')
    ORDER BY type, state_key, state_group DESC
) UNION (
    SELECT DISTINCT ON (type, state_key)
        state_group, type, state_key, event_id
    FROM state_groups_state
    INNER JOIN sgs USING (state_group)
    WHERE (type = 'm.room.history_visibility' AND state_key = '')
    ORDER BY type, state_key, state_group DESC
);
                                                                                                QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=1272.63..1272.66 rows=2 width=104) (actual time=1.710..1.719 rows=2 loops=1)
   CTE sgs
     ->  Recursive Union  (cost=0.00..284.28 rows=101 width=8) (actual time=0.003..0.623 rows=59 loops=1)
           ->  Result  (cost=0.00..0.01 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=1)
           ->  Nested Loop  (cost=0.57..28.23 rows=10 width=8) (actual time=0.008..0.009 rows=1 loops=59)
                 ->  WorkTable Scan on sgs s  (cost=0.00..0.20 rows=10 width=8) (actual time=0.000..0.000 rows=1 loops=59)
                 ->  Index Only Scan using state_group_edges_unique_idx on state_group_edges e  (cost=0.57..2.79 rows=1 width=16) (actual time=0.007..0.008 rows=1 loops=59)
                       Index Cond: (state_group = s.state_group)
                       Heap Fetches: 58
   ->  Sort  (cost=988.35..988.36 rows=2 width=104) (actual time=1.709..1.712 rows=2 loops=1)
         Sort Key: state_groups_state.state_group, state_groups_state.type, state_groups_state.state_key, state_groups_state.event_id
         Sort Method: quicksort  Memory: 25kB
         ->  Append  (cost=494.15..988.34 rows=2 width=104) (actual time=1.253..1.702 rows=2 loops=1)
               ->  Unique  (cost=494.15..494.16 rows=1 width=87) (actual time=1.252..1.254 rows=1 loops=1)
                     ->  Sort  (cost=494.15..494.16 rows=1 width=87) (actual time=1.251..1.252 rows=1 loops=1)
                           Sort Key: state_groups_state.state_group DESC
                           Sort Method: quicksort  Memory: 25kB
                           ->  Nested Loop  (cost=0.84..494.14 rows=1 width=87) (actual time=1.232..1.245 rows=1 loops=1)
                                 ->  CTE Scan on sgs  (cost=0.00..2.02 rows=101 width=8) (actual time=0.006..0.656 rows=59 loops=1)
                                 ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.84..4.86 rows=1 width=87) (actual time=0.009..0.009 rows=0 loops=59)
                                       Index Cond: ((state_group = sgs.state_group) AND (type = 'm.room.member'::text) AND (state_key = '@madlittlemods:matrix.org'::text))
               ->  Unique  (cost=494.15..494.16 rows=1 width=87) (actual time=0.444..0.445 rows=1 loops=1)
                     ->  Sort  (cost=494.15..494.16 rows=1 width=87) (actual time=0.443..0.444 rows=1 loops=1)
                           Sort Key: state_groups_state_1.state_group DESC
                           Sort Method: quicksort  Memory: 25kB
                           ->  Nested Loop  (cost=0.84..494.14 rows=1 width=87) (actual time=0.438..0.440 rows=1 loops=1)
                                 ->  CTE Scan on sgs sgs_1  (cost=0.00..2.02 rows=101 width=8) (actual time=0.001..0.014 rows=59 loops=1)
debug2: channel 0: window 997859 sent adjust 50717
                                 ->  Index Scan using state_groups_state_type_idx on state_groups_state state_groups_state_1  (cost=0.84..4.86 rows=1 width=87) (actual time=0.007..0.007 rows=0 loops=59)
                                       Index Cond: ((state_group = sgs_1.state_group) AND (type = 'm.room.history_visibility'::text) AND (state_key = ''::text))
 Planning Time: 4.756 ms
 Execution Time: 1.814 ms
(31 rows)

Time: 7.538 ms

Dev notes

Previous optimizations:


Grabbing all of the state via get_state_for_events is the longest part of filter_events_for_client

  • filter_events_for_client
    • get_state_for_events
      • get_state_group_for_events
        • _get_state_group_for_events
      • _get_state_for_groups
        • _get_state_for_groups_using_cache
        • _get_state_groups_from_groups 🐢
          • _get_state_groups_from_groups_txn (db._get_state_groups_from_groups)
      • get_events
        • ...

Of all the state we grab in filter_events_for_client, we only ever use EventTypes.RoomHistoryVisibility and EventTypes.Member from it

  • filter_events_for_client
  • _check_client_allowed_to_see_event
    • _check_filter_send_to_client
      • ->Uses no state
    • get_effective_room_visibility_from_state
      • -> EventTypes.RoomHistoryVisibility 🎈
    • _check_history_visibility
      • -> Uses visibility result from above
    • _check_membership
      • -> Uses visibility result from above
      • -> EventTypes.Member 🎈

Relevant database tables:

  • state_groups
  • state_groups_state
  • state_group_edges
  • event_to_state_groups

Docs:

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Pull request includes a sign off
  • Code style is correct
    (run the linters)

Seems like it will make it a magnitude faster.

Fix #14108

Part of #13356 (comment)
@MadLittleMods MadLittleMods added A-Performance Performance, both client-facing and admin-facing T-Enhancement New features, changes in functionality, improvements in performance, or user-facing enhancements. A-Messages-Endpoint /messages client API endpoint (`RoomMessageListRestServlet`) (which also triggers /backfill) labels Nov 22, 2022
@MadLittleMods MadLittleMods changed the title Optimize filter_events_for_client for faster /messages Optimize filter_events_for_client for faster /messages - v2 Nov 22, 2022
@MadLittleMods MadLittleMods marked this pull request as ready for review November 22, 2022 20:41
@MadLittleMods MadLittleMods requested a review from a team as a code owner November 22, 2022 20:41
@erikjohnston erikjohnston merged commit 7f78b38 into develop Nov 22, 2022
@erikjohnston erikjohnston deleted the madlittlemods/14108-optimize-filter_events_for_client-v2 branch November 22, 2022 21:56
@MadLittleMods
Copy link
Contributor Author

Thanks for the review @DMRobertson and @erikjohnston 🐡

@MadLittleMods
Copy link
Contributor Author

MadLittleMods commented Nov 23, 2022

For my benchmark /messages?dir=b&limit=500 request against Matrix HQ, we've reduced the filter_events_for_client part of the request from ~8s to ~1s.

Looking at the graphs, shows an even better picture where we have improved /messages timing across the board on matrix.org (not just in requests against large rooms or with high limits):

https://grafana.matrix.org/d/dYoRgTgVz/messages-timing?orgId=1&from=1668971225671&to=1669230425671

Improved /messages response time Apdex. We can see the obviously settling down in the blue line for large rooms (>1000 members)

But Apdex also improved across the board regardless of room size when we focus it specifically:

@erikjohnston also saw that we reduced the table block reads from disk for state_groups_state table in the database

https://grafana.matrix.org/d/000000009/postgres?orgId=1&refresh=1m&var-data_source=Prometheus&var-database=matrix&var-bucket_size=%24__auto_interval_bucket_size&var-instance=manticore.matrix.org%3A9113&from=1668971225671&to=1669230425671

There isn't a big obvious an obvious reduction jump in _get_state_groups_from_groups transaction time as I would expect 🤔 although squinting at it, seems to show improvement.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Dec 12, 2022
Synapse 1.73.0 (2022-12-06)
===========================

Please note that legacy Prometheus metric names have been removed in this release; see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.73/docs/upgrade.md#legacy-prometheus-metric-names-have-now-been-removed) for more details.

Features
--------

- Speed-up `/messages` with `filter_events_for_client`
  optimizations. ([\#14527](matrix-org/synapse#14527))
- Improve DB performance by reducing amount of data that gets read in
  `device_lists_changes_in_room`. ([\#14534](matrix-org/synapse#14534))
- Adds support for handling avatar in SSO OIDC login. Contributed by
  @ashfame. ([\#13917](matrix-org/synapse#13917))
- Move MSC3030 `/timestamp_to_event` endpoints to stable `v1` location
  (`/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`,
  `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`). ([\#14471](matrix-org/synapse#14471))
- Reduce database load of [Client-Server
  endpoints](https://spec.matrix.org/v1.5/client-server-api/#aggregations)
  which return bundled
  aggregations. ([\#14491](matrix-org/synapse#14491),
  [\#14508](matrix-org/synapse#14508),
  [\#14510](matrix-org/synapse#14510))
- Add unstable support for an Extensible Events room version
  (`org.matrix.msc1767.10`) via
  [MSC1767](matrix-org/matrix-spec-proposals#1767),
  [MSC3931](matrix-org/matrix-spec-proposals#3931),
  [MSC3932](matrix-org/matrix-spec-proposals#3932),
  and
  [MSC3933](matrix-org/matrix-spec-proposals#3933).
  ([\#14520](matrix-org/synapse#14520),
  [\#14521](matrix-org/synapse#14521),
  [\#14524](matrix-org/synapse#14524))
- Prune user's old devices on login if they have too
  many. ([\#14038](matrix-org/synapse#14038),
  [\#14580](matrix-org/synapse#14580))

Deprecations and Removals
-------------------------

- Remove legacy Prometheus metrics names. They were deprecated in
  Synapse v1.69.0 and disabled by default in Synapse
  v1.71.0. ([\#14538](matrix-org/synapse#14538))
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Dec 15, 2022
Synapse 1.73.0 (2022-12-06)
===========================

Please note that legacy Prometheus metric names have been removed in this release; see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.73/docs/upgrade.md#legacy-prometheus-metric-names-have-now-been-removed) for more details.

No significant changes since 1.73.0rc2.

Synapse 1.73.0rc2 (2022-12-01)
==============================

Bugfixes
--------

- Fix a regression in Synapse 1.73.0rc1 where Synapse's main process would stop responding to HTTP requests when a user with a large number of devices logs in. ([\matrix-org#14582](matrix-org#14582))

Synapse 1.73.0rc1 (2022-11-29)
==============================

Features
--------

- Speed-up `/messages` with `filter_events_for_client` optimizations. ([\matrix-org#14527](matrix-org#14527))
- Improve DB performance by reducing amount of data that gets read in `device_lists_changes_in_room`. ([\matrix-org#14534](matrix-org#14534))
- Adds support for handling avatar in SSO OIDC login. Contributed by @ashfame. ([\matrix-org#13917](matrix-org#13917))
- Move MSC3030 `/timestamp_to_event` endpoints to stable `v1` location (`/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`, `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`). ([\matrix-org#14471](matrix-org#14471))
- Reduce database load of [Client-Server endpoints](https://spec.matrix.org/v1.5/client-server-api/#aggregations) which return bundled aggregations. ([\matrix-org#14491](matrix-org#14491), [\matrix-org#14508](matrix-org#14508), [\matrix-org#14510](matrix-org#14510))
- Add unstable support for an Extensible Events room version (`org.matrix.msc1767.10`) via [MSC1767](matrix-org/matrix-spec-proposals#1767), [MSC3931](matrix-org/matrix-spec-proposals#3931), [MSC3932](matrix-org/matrix-spec-proposals#3932), and [MSC3933](matrix-org/matrix-spec-proposals#3933). ([\matrix-org#14520](matrix-org#14520), [\matrix-org#14521](matrix-org#14521), [\matrix-org#14524](matrix-org#14524))
- Prune user's old devices on login if they have too many. ([\matrix-org#14038](matrix-org#14038), [\matrix-org#14580](matrix-org#14580))

Bugfixes
--------

- Fix a long-standing bug where paginating from the start of a room did not work. Contributed by @gnunicorn. ([\matrix-org#14149](matrix-org#14149))
- Fix a bug introduced in Synapse 1.58.0 where a user with presence state `org.matrix.msc3026.busy` would mistakenly be set to `online` when calling `/sync` or `/events` on a worker process. ([\matrix-org#14393](matrix-org#14393))
- Fix a bug introduced in Synapse 1.70.0 where a receipt's thread ID was not sent over federation. ([\matrix-org#14466](matrix-org#14466))
- Fix a long-standing bug where the [List media admin API](https://matrix-org.github.io/synapse/latest/admin_api/media_admin_api.html#list-all-media-in-a-room) would fail when processing an image with broken thumbnail information. ([\matrix-org#14537](matrix-org#14537))
- Fix a bug introduced in Synapse 1.67.0 where two logging context warnings would be logged on startup. ([\matrix-org#14574](matrix-org#14574))
- In application service transactions that include the experimental `org.matrix.msc3202.device_one_time_key_counts` key, include a duplicate key of `org.matrix.msc3202.device_one_time_keys_count` to match the name proposed by [MSC3202](matrix-org/matrix-spec-proposals#3202). ([\matrix-org#14565](matrix-org#14565))
- Fix a bug introduced in Synapse 0.9 where Synapse would fail to fetch server keys whose IDs contain a forward slash. ([\matrix-org#14490](matrix-org#14490))

Improved Documentation
----------------------

- Fixed link to 'Synapse administration endpoints'. ([\matrix-org#14499](matrix-org#14499))

Deprecations and Removals
-------------------------

- Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. ([\matrix-org#14538](matrix-org#14538))

Internal Changes
----------------

- Improve type hinting throughout Synapse. ([\matrix-org#14055](matrix-org#14055), [\matrix-org#14412](matrix-org#14412), [\matrix-org#14529](matrix-org#14529), [\matrix-org#14452](matrix-org#14452)).
- Remove old stream ID tracking code. Contributed by Nick @beeper (@Fizzadar). ([\matrix-org#14376](matrix-org#14376), [\matrix-org#14468](matrix-org#14468))
- Remove the `worker_main_http_uri` configuration setting. This is now handled via internal replication. ([\matrix-org#14400](matrix-org#14400), [\matrix-org#14476](matrix-org#14476))
- Refactor `federation_sender` and `pusher` configuration loading. ([\matrix-org#14496](matrix-org#14496))
([\matrix-org#14509](matrix-org#14509), [\matrix-org#14573](matrix-org#14573))
- Faster joins: do not wait for full state when creating events to send. ([\matrix-org#14403](matrix-org#14403))
- Faster joins: filter out non local events when a room doesn't have its full state. ([\matrix-org#14404](matrix-org#14404))
- Faster joins: send events to initial list of servers if we don't have the full state yet. ([\matrix-org#14408](matrix-org#14408))
- Faster joins: use servers list approximation received during `send_join` (potentially updated with received membership events) in `assert_host_in_room`. ([\matrix-org#14515](matrix-org#14515))
- Fix type logic in TCP replication code that prevented correctly ignoring blank commands. ([\matrix-org#14449](matrix-org#14449))
- Remove option to skip locking of tables when performing emulated upserts, to avoid a class of bugs in future. ([\matrix-org#14469](matrix-org#14469))
- `scripts-dev/federation_client`: Fix routing on servers with `.well-known` files. ([\matrix-org#14479](matrix-org#14479))
- Reduce default third party invite rate limit to 216 invites per day. ([\matrix-org#14487](matrix-org#14487))
- Refactor conversion of device list changes in room to outbound pokes to track unconverted rows using a `(stream ID, room ID)` position instead of updating the `converted_to_destinations` flag on every row. ([\matrix-org#14516](matrix-org#14516))
- Add more prompts to the bug report form. ([\matrix-org#14522](matrix-org#14522))
- Extend editorconfig rules on indent and line length to `.pyi` files. ([\matrix-org#14526](matrix-org#14526))
- Run Rust CI when `Cargo.lock` changes. This is particularly useful for dependabot updates. ([\matrix-org#14571](matrix-org#14571))
- Fix a possible variable shadow in `create_new_client_event`. ([\matrix-org#14575](matrix-org#14575))
- Bump various dependencies in the `poetry.lock` file and in CI scripts. ([\matrix-org#14557](matrix-org#14557), [\matrix-org#14559](matrix-org#14559), [\matrix-org#14560](matrix-org#14560), [\matrix-org#14500](matrix-org#14500), [\matrix-org#14501](matrix-org#14501), [\matrix-org#14502](matrix-org#14502), [\matrix-org#14503](matrix-org#14503), [\matrix-org#14504](matrix-org#14504), [\matrix-org#14505](matrix-org#14505)).

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmOPLnYACgkQLS76LzL7
# 4Edwpg/+KXpg2ZdiJ0Yaly9VHVeiqdHRi5D7WPS6n8YBsdRx9EQHzOBkD5HAW8hE
# oz0c+zDS01ORlEWD825NYXjgaE1ijtZFvGxsftYTVuTYlVRR2m+r9jhDv9pVHT53
# TKtQVKpG0IUsuyukRBrweDcEeO0MA0nGpvaaQUhmftzWgy4yD3AjZyIgx0Ckg8pg
# OwgrzGqA7FQs4MEeOxmk1H39fZg4dlo4nmI4whvAodgaGeS9sU8t+3Qj4PVod8v/
# AkVesJcruaTHuVMb+Xp8JKezb09SsIR94gmHalC5sL+41+6XAy9BtQ/cRDfCReG3
# U1I1x1h1+EQjTP6XzMmjQHLbfI2gUJBC4I2p3e2gZ4cMm9rVz94R1dBiRk8ZgRIC
# cJFD9BvaAtb2PSTvyFBoHsrrn/u12i8fYFWu4Z4rO6dOGI83dZHeZzVw4UsVeqIK
# 5+njQwcwQsrwL3AKLjbbdqmbmhXcF6LchIK2L+NuuvdiOfvXvkO0bdjBryVEbMqB
# IOtAAWzwYaoUwVucMbBtXt/EqQS7biGkbDxsL8CDvaBwM/JSsUWXBafsV1FmxF2A
# q6KAeKpfelefoegosTYD0Md+l39xdF8Z19XaKV3GeHZEY+HE3RJXJm+Pa8SJ+IF8
# Y1od9cB/H+fYSsWCWj1OJNqTIAozh6f1Pe2nFuFDxdBwABXc/pg=
# =IBEL
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Dec  6 11:58:46 2022 GMT
# gpg:                using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047
# gpg: Can't check signature: No public key

# Conflicts:
#	poetry.lock
#	synapse/push/bulk_push_rule_evaluator.py
#	synapse/storage/databases/main/account_data.py
#	synapse/storage/databases/main/receipts.py
realtyem added a commit to realtyem/synapse-unraid that referenced this pull request Dec 18, 2022
Synapse 1.73.0 (2022-12-06)
===========================

Please note that legacy Prometheus metric names have been removed in this release; see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.73/docs/upgrade.md#legacy-prometheus-metric-names-have-now-been-removed) for more details.

No significant changes since 1.73.0rc2.

Synapse 1.73.0rc2 (2022-12-01)
==============================

Bugfixes
--------

- Fix a regression in Synapse 1.73.0rc1 where Synapse's main process would stop responding to HTTP requests when a user with a large number of devices logs in. ([\#14582](matrix-org/synapse#14582))

Synapse 1.73.0rc1 (2022-11-29)
==============================

Features
--------

- Speed-up `/messages` with `filter_events_for_client` optimizations. ([\#14527](matrix-org/synapse#14527))
- Improve DB performance by reducing amount of data that gets read in `device_lists_changes_in_room`. ([\#14534](matrix-org/synapse#14534))
- Adds support for handling avatar in SSO OIDC login. Contributed by @ashfame. ([\#13917](matrix-org/synapse#13917))
- Move MSC3030 `/timestamp_to_event` endpoints to stable `v1` location (`/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`, `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`). ([\#14471](matrix-org/synapse#14471))
- Reduce database load of [Client-Server endpoints](https://spec.matrix.org/v1.5/client-server-api/#aggregations) which return bundled aggregations. ([\#14491](matrix-org/synapse#14491), [\#14508](matrix-org/synapse#14508), [\#14510](matrix-org/synapse#14510))
- Add unstable support for an Extensible Events room version (`org.matrix.msc1767.10`) via [MSC1767](matrix-org/matrix-spec-proposals#1767), [MSC3931](matrix-org/matrix-spec-proposals#3931), [MSC3932](matrix-org/matrix-spec-proposals#3932), and [MSC3933](matrix-org/matrix-spec-proposals#3933). ([\#14520](matrix-org/synapse#14520), [\#14521](matrix-org/synapse#14521), [\#14524](matrix-org/synapse#14524))
- Prune user's old devices on login if they have too many. ([\#14038](matrix-org/synapse#14038), [\#14580](matrix-org/synapse#14580))

Bugfixes
--------

- Fix a long-standing bug where paginating from the start of a room did not work. Contributed by @gnunicorn. ([\#14149](matrix-org/synapse#14149))
- Fix a bug introduced in Synapse 1.58.0 where a user with presence state `org.matrix.msc3026.busy` would mistakenly be set to `online` when calling `/sync` or `/events` on a worker process. ([\#14393](matrix-org/synapse#14393))
- Fix a bug introduced in Synapse 1.70.0 where a receipt's thread ID was not sent over federation. ([\#14466](matrix-org/synapse#14466))
- Fix a long-standing bug where the [List media admin API](https://matrix-org.github.io/synapse/latest/admin_api/media_admin_api.html#list-all-media-in-a-room) would fail when processing an image with broken thumbnail information. ([\#14537](matrix-org/synapse#14537))
- Fix a bug introduced in Synapse 1.67.0 where two logging context warnings would be logged on startup. ([\#14574](matrix-org/synapse#14574))
- In application service transactions that include the experimental `org.matrix.msc3202.device_one_time_key_counts` key, include a duplicate key of `org.matrix.msc3202.device_one_time_keys_count` to match the name proposed by [MSC3202](matrix-org/matrix-spec-proposals#3202). ([\#14565](matrix-org/synapse#14565))
- Fix a bug introduced in Synapse 0.9 where Synapse would fail to fetch server keys whose IDs contain a forward slash. ([\#14490](matrix-org/synapse#14490))

Improved Documentation
----------------------

- Fixed link to 'Synapse administration endpoints'. ([\#14499](matrix-org/synapse#14499))

Deprecations and Removals
-------------------------

- Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. ([\#14538](matrix-org/synapse#14538))

Internal Changes
----------------

- Improve type hinting throughout Synapse. ([\#14055](matrix-org/synapse#14055), [\#14412](matrix-org/synapse#14412), [\#14529](matrix-org/synapse#14529), [\#14452](matrix-org/synapse#14452)).
- Remove old stream ID tracking code. Contributed by Nick @beeper (@Fizzadar). ([\#14376](matrix-org/synapse#14376), [\#14468](matrix-org/synapse#14468))
- Remove the `worker_main_http_uri` configuration setting. This is now handled via internal replication. ([\#14400](matrix-org/synapse#14400), [\#14476](matrix-org/synapse#14476))
- Refactor `federation_sender` and `pusher` configuration loading. ([\#14496](matrix-org/synapse#14496))
([\#14509](matrix-org/synapse#14509), [\#14573](matrix-org/synapse#14573))
- Faster joins: do not wait for full state when creating events to send. ([\#14403](matrix-org/synapse#14403))
- Faster joins: filter out non local events when a room doesn't have its full state. ([\#14404](matrix-org/synapse#14404))
- Faster joins: send events to initial list of servers if we don't have the full state yet. ([\#14408](matrix-org/synapse#14408))
- Faster joins: use servers list approximation received during `send_join` (potentially updated with received membership events) in `assert_host_in_room`. ([\#14515](matrix-org/synapse#14515))
- Fix type logic in TCP replication code that prevented correctly ignoring blank commands. ([\#14449](matrix-org/synapse#14449))
- Remove option to skip locking of tables when performing emulated upserts, to avoid a class of bugs in future. ([\#14469](matrix-org/synapse#14469))
- `scripts-dev/federation_client`: Fix routing on servers with `.well-known` files. ([\#14479](matrix-org/synapse#14479))
- Reduce default third party invite rate limit to 216 invites per day. ([\#14487](matrix-org/synapse#14487))
- Refactor conversion of device list changes in room to outbound pokes to track unconverted rows using a `(stream ID, room ID)` position instead of updating the `converted_to_destinations` flag on every row. ([\#14516](matrix-org/synapse#14516))
- Add more prompts to the bug report form. ([\#14522](matrix-org/synapse#14522))
- Extend editorconfig rules on indent and line length to `.pyi` files. ([\#14526](matrix-org/synapse#14526))
- Run Rust CI when `Cargo.lock` changes. This is particularly useful for dependabot updates. ([\#14571](matrix-org/synapse#14571))
- Fix a possible variable shadow in `create_new_client_event`. ([\#14575](matrix-org/synapse#14575))
- Bump various dependencies in the `poetry.lock` file and in CI scripts. ([\#14557](matrix-org/synapse#14557), [\#14559](matrix-org/synapse#14559), [\#14560](matrix-org/synapse#14560), [\#14500](matrix-org/synapse#14500), [\#14501](matrix-org/synapse#14501), [\#14502](matrix-org/synapse#14502), [\#14503](matrix-org/synapse#14503), [\#14504](matrix-org/synapse#14504), [\#14505](matrix-org/synapse#14505)).
@MadLittleMods MadLittleMods added the A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db label May 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-Messages-Endpoint /messages client API endpoint (`RoomMessageListRestServlet`) (which also triggers /backfill) A-Performance Performance, both client-facing and admin-facing T-Enhancement New features, changes in functionality, improvements in performance, or user-facing enhancements.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

filter_events_for_client is time-consuming and expensive
3 participants