Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add support for MSC3202: sending one-time key counts and fallback key usage states to Application Services. #11617

Merged
merged 26 commits into from
Feb 24, 2022
Merged

Add support for MSC3202: sending one-time key counts and fallback key usage states to Application Services. #11617

merged 26 commits into from
Feb 24, 2022

Conversation

reivilibre
Copy link
Contributor

@reivilibre reivilibre commented Dec 20, 2021 •
edited
Loading

It's difficult to divide this one up any further (apart from minor things ~ 10 lines for config, etc, or leaving things unimplemented as stubs); if you can see a way that would do it effectively then let me know and I'll try to do that.

The general idea here is:

  • Application Services will now receive one-time key counts and fallback key usage states for their users' devices
  • The idea we settled on in Encrypted appservices: how to? #10653 is as follows:
    • Compute the set of 'interesting users' to the application service for this transaction, hereby defined as:
      • the AS's users that are in rooms which receive an event in this transaction; and
      • the Application Service sender_localpart (unsure if this has a more proper name, but it's vaguely the 'main user ID' for the application service and is important enough that it should not miss OTK/FBK changes)
    • Retrieve counts / FBK states for all of those and send them out

As discussed in #10653, doing it this way is easier implementation-wise (since otherwise we would need to somehow track changes) and is chattier than needed.

This should largely be a commit-by-commit review, though there are a few 'boring' commits.

MSC3202 is available here.

@reivilibre reivilibre changed the base branch from develop to anoa/e2e_as_to_device December 20, 2021 16:14
Base automatically changed from anoa/e2e_as_to_device to develop February 1, 2022 14:13
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch 2 times, most recently from bfa0a9d to 23ec85f Compare February 4, 2022 10:04
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch from 23ec85f to 708b3d7 Compare February 4, 2022 12:41
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch 2 times, most recently from d036dac to 369a0bb Compare February 8, 2022 14:22
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch from 369a0bb to 5a3a5d9 Compare February 8, 2022 15:01
@reivilibre reivilibre marked this pull request as ready for review February 9, 2022 11:20
@reivilibre reivilibre requested a review from a team as a code owner February 9, 2022 11:20
erikjohnston
erikjohnston reviewed Feb 11, 2022
View reviewed changes
synapse/appservice/api.py Outdated Show resolved Hide resolved
synapse/appservice/scheduler.py Outdated Show resolved Hide resolved
synapse/config/experimental.py
# sending one-time key counts and fallback key usage to application services.
self.msc3202_transaction_extensions: bool = experimental.get(
"msc3202_transaction_extensions", False
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth having both options?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge if you think that's best — I wasn't right sure because they're really quite separate features that could easily have been separate MSCs, but not fussed either way.

synapse/storage/databases/main/appservice.py Outdated Show resolved Hide resolved
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch from b221c0d to f059d46 Compare February 11, 2022 11:40
@reivilibre reivilibre force-pushed the rei/msc3202_otks_fbks branch from f059d46 to c696a03 Compare February 11, 2022 12:33
erikjohnston
erikjohnston reviewed Feb 14, 2022
View reviewed changes
Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM

synapse/appservice/scheduler.py Outdated Show resolved Hide resolved
synapse/storage/databases/main/end_to_end_keys.py Show resolved Hide resolved
synapse/storage/databases/main/end_to_end_keys.py Show resolved Hide resolved
@reivilibre reivilibre enabled auto-merge (squash) February 24, 2022 17:24
@reivilibre reivilibre merged commit 2cc5ea9 into develop Feb 24, 2022
@reivilibre reivilibre deleted the rei/msc3202_otks_fbks branch February 24, 2022 17:55
reivilibre added a commit that referenced this pull request Mar 2, 2022
@reivilibre
Merge tag 'v1.54.0rc1' into develop
594a07e 
Synapse 1.54.0rc1 (2022-03-02)
==============================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](#12001), [\#12067](#12067))
- Enable modules to set a custom display name when registering a user. ([\#12009](#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](#12020), ([\#12022](#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\#12089](#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](#11997))
- Use Python 3.9 in Docker images by default. ([\#12112](#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](#11599))
- Explain the meaning of spam checker callbacks' return values. ([\#12003](#12003))
- Clarify information about external Identity Provider IDs. ([\#12004](#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](#11900))
- Add tests for device list changes between local users. ([\#11972](#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\#11974](#11974))
- Add missing type hints to storage classes. ([\#11984](#11984))
- Refactor the search code for improved readability. ([\#11991](#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](#11994))
- Limit concurrent joins from applications services. ([\#11996](#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](#12005), [\#12039](#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](#12025))
- Upgrade Mypy to version 0.931. ([\#12030](#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\#12031](#12031), [\#12070](#12070))
- Minor typing fixes. ([\#12034](#12034), [\#12069](#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](#12051))
- Move configuration out of `setup.cfg`. ([\#12052](#12052), [\#12059](#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\#12060](#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](#12063))
- Add type hints to `tests/rest/client`. ([\#12066](#12066), [\#12072](#12072), [\#12084](#12084), [\#12094](#12094))
- Add some logging to `/sync` to try and track down #11916. ([\#12068](#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](#12106))
- Improve exception handling for concurrent execution. ([\#12109](#12109))
- Advertise support for Python 3.10 in packaging files. ([\#12111](#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](#12119))
Half-Shot
Half-Shot reviewed Apr 21, 2022
View reviewed changes
synapse/appservice/api.py
] = one_time_key_counts
if unused_fallback_keys:
body[
"org.matrix.msc3202.device_unused_fallback_keys"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having tried to play with this just now, I think this is wrong. The MSC states device_unused_fallback_key_types

Fizzadar added a commit to Fizzadar/synapse that referenced this pull request Apr 25, 2022
@Fizzadar
Merge tag 'v1.54.0' into beeper-master
c7d5eda 
Synapse 1.54.0 (2022-03-08)
===========================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Bugfixes
--------

- Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\matrix-org#12141](matrix-org#12141))
- Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\matrix-org#12129](matrix-org#12129), [\matrix-org#12177](matrix-org#12177))

Internal Changes
----------------

- Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\matrix-org#12127](matrix-org#12127))
- Relax the version guard for "packaging" added in [\matrix-org#12088](matrix-org#12088). ([\matrix-org#12166](matrix-org#12166))

Synapse 1.54.0rc1 (2022-03-02)
==============================

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\matrix-org#11617](matrix-org#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\matrix-org#11985](matrix-org#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\matrix-org#12000](matrix-org#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\matrix-org#12001](matrix-org#12001), [\matrix-org#12067](matrix-org#12067))
- Enable modules to set a custom display name when registering a user. ([\matrix-org#12009](matrix-org#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\matrix-org#12020](matrix-org#12020), ([\matrix-org#12022](matrix-org#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\matrix-org#12021](matrix-org#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\matrix-org#12058](matrix-org#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\matrix-org#12062](matrix-org#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\matrix-org#11992](matrix-org#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\matrix-org#11999](matrix-org#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\matrix-org#12024](matrix-org#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\matrix-org#12037](matrix-org#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\matrix-org#12056](matrix-org#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\matrix-org#12077](matrix-org#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\matrix-org#12089](matrix-org#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\matrix-org#12098](matrix-org#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\matrix-org#12100](matrix-org#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\matrix-org#12105](matrix-org#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\matrix-org#11835](matrix-org#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\matrix-org#11997](matrix-org#11997))
- Use Python 3.9 in Docker images by default. ([\matrix-org#12112](matrix-org#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\matrix-org#11599](matrix-org#11599))
- Explain the meaning of spam checker callbacks' return values. ([\matrix-org#12003](matrix-org#12003))
- Clarify information about external Identity Provider IDs. ([\matrix-org#12004](matrix-org#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\matrix-org#11865](matrix-org#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\matrix-org#12008](matrix-org#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\matrix-org#12018](matrix-org#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\matrix-org#12073](matrix-org#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\matrix-org#11808](matrix-org#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\matrix-org#11900](matrix-org#11900))
- Add tests for device list changes between local users. ([\matrix-org#11972](matrix-org#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\matrix-org#11974](matrix-org#11974))
- Add missing type hints to storage classes. ([\matrix-org#11984](matrix-org#11984))
- Refactor the search code for improved readability. ([\matrix-org#11991](matrix-org#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\matrix-org#11994](matrix-org#11994))
- Limit concurrent joins from applications services. ([\matrix-org#11996](matrix-org#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\matrix-org#12005](matrix-org#12005), [\matrix-org#12039](matrix-org#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\matrix-org#12011](matrix-org#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\matrix-org#12012](matrix-org#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\matrix-org#12013](matrix-org#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\matrix-org#12015](matrix-org#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\matrix-org#12016](matrix-org#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\matrix-org#12019](matrix-org#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\matrix-org#12025](matrix-org#12025))
- Upgrade Mypy to version 0.931. ([\matrix-org#12030](matrix-org#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\matrix-org#12031](matrix-org#12031), [\matrix-org#12070](matrix-org#12070))
- Minor typing fixes. ([\matrix-org#12034](matrix-org#12034), [\matrix-org#12069](matrix-org#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\matrix-org#12041](matrix-org#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\matrix-org#12051](matrix-org#12051))
- Move configuration out of `setup.cfg`. ([\matrix-org#12052](matrix-org#12052), [\matrix-org#12059](matrix-org#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\matrix-org#12060](matrix-org#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\matrix-org#12063](matrix-org#12063))
- Add type hints to `tests/rest/client`. ([\matrix-org#12066](matrix-org#12066), [\matrix-org#12072](matrix-org#12072), [\matrix-org#12084](matrix-org#12084), [\matrix-org#12094](matrix-org#12094))
- Add some logging to `/sync` to try and track down matrix-org#11916. ([\matrix-org#12068](matrix-org#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\matrix-org#12088](matrix-org#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\matrix-org#12092](matrix-org#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\matrix-org#12099](matrix-org#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\matrix-org#12106](matrix-org#12106))
- Improve exception handling for concurrent execution. ([\matrix-org#12109](matrix-org#12109))
- Advertise support for Python 3.10 in packaging files. ([\matrix-org#12111](matrix-org#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\matrix-org#12119](matrix-org#12119))
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Apr 28, 2022
@babolivier
Merge tag 'v1.54.0' into babolivier/dinsic_1.58
d721bf6 
Synapse 1.54.0 (2022-03-08)
===========================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Bugfixes
--------

- Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\#12141](matrix-org/synapse#12141))
- Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\#12129](matrix-org/synapse#12129), [\#12177](matrix-org/synapse#12177))

Internal Changes
----------------

- Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\#12127](matrix-org/synapse#12127))
- Relax the version guard for "packaging" added in [\#12088](matrix-org/synapse#12088). ([\#12166](matrix-org/synapse#12166))

Synapse 1.54.0rc1 (2022-03-02)
==============================

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](matrix-org/synapse#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](matrix-org/synapse#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](matrix-org/synapse#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](matrix-org/synapse#12001), [\#12067](matrix-org/synapse#12067))
- Enable modules to set a custom display name when registering a user. ([\#12009](matrix-org/synapse#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](matrix-org/synapse#12020), ([\#12022](matrix-org/synapse#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](matrix-org/synapse#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](matrix-org/synapse#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](matrix-org/synapse#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](matrix-org/synapse#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](matrix-org/synapse#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](matrix-org/synapse#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](matrix-org/synapse#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](matrix-org/synapse#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](matrix-org/synapse#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\#12089](matrix-org/synapse#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](matrix-org/synapse#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](matrix-org/synapse#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](matrix-org/synapse#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](matrix-org/synapse#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](matrix-org/synapse#11997))
- Use Python 3.9 in Docker images by default. ([\#12112](matrix-org/synapse#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](matrix-org/synapse#11599))
- Explain the meaning of spam checker callbacks' return values. ([\#12003](matrix-org/synapse#12003))
- Clarify information about external Identity Provider IDs. ([\#12004](matrix-org/synapse#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](matrix-org/synapse#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](matrix-org/synapse#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](matrix-org/synapse#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](matrix-org/synapse#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](matrix-org/synapse#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](matrix-org/synapse#11900))
- Add tests for device list changes between local users. ([\#11972](matrix-org/synapse#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\#11974](matrix-org/synapse#11974))
- Add missing type hints to storage classes. ([\#11984](matrix-org/synapse#11984))
- Refactor the search code for improved readability. ([\#11991](matrix-org/synapse#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](matrix-org/synapse#11994))
- Limit concurrent joins from applications services. ([\#11996](matrix-org/synapse#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](matrix-org/synapse#12005), [\#12039](matrix-org/synapse#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](matrix-org/synapse#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](matrix-org/synapse#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](matrix-org/synapse#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](matrix-org/synapse#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](matrix-org/synapse#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](matrix-org/synapse#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](matrix-org/synapse#12025))
- Upgrade Mypy to version 0.931. ([\#12030](matrix-org/synapse#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\#12031](matrix-org/synapse#12031), [\#12070](matrix-org/synapse#12070))
- Minor typing fixes. ([\#12034](matrix-org/synapse#12034), [\#12069](matrix-org/synapse#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](matrix-org/synapse#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](matrix-org/synapse#12051))
- Move configuration out of `setup.cfg`. ([\#12052](matrix-org/synapse#12052), [\#12059](matrix-org/synapse#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\#12060](matrix-org/synapse#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](matrix-org/synapse#12063))
- Add type hints to `tests/rest/client`. ([\#12066](matrix-org/synapse#12066), [\#12072](matrix-org/synapse#12072), [\#12084](matrix-org/synapse#12084), [\#12094](matrix-org/synapse#12094))
- Add some logging to `/sync` to try and track down #11916. ([\#12068](matrix-org/synapse#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](matrix-org/synapse#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](matrix-org/synapse#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](matrix-org/synapse#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](matrix-org/synapse#12106))
- Improve exception handling for concurrent execution. ([\#12109](matrix-org/synapse#12109))
- Advertise support for Python 3.10 in packaging files. ([\#12111](matrix-org/synapse#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](matrix-org/synapse#12119))
This was referenced Jul 11, 2022
Merged
Merged
@turt2live turt2live mentioned this pull request Jul 15, 2022
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@Half-Shot Half-Shot Half-Shot left review comments

@erikjohnston erikjohnston erikjohnston approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@reivilibre @Half-Shot @erikjohnston