Sliding sync: Store the per-connection state in the database. #17599

erikjohnston · 2024-08-22T16:20:54Z

Based on #17600

synapse/storage/schema/main/delta/87/02_per_connection_state.sql

tests/rest/client/sliding_sync/test_rooms_required_state.py

MadLittleMods · 2024-08-26T15:56:16Z

synapse/storage/schema/__init__.py

@@ -19,7 +19,7 @@
 #
 #

-SCHEMA_VERSION = 86  # remember to update the list below when updating
+SCHEMA_VERSION = 87  # remember to update the list below when updating


remember to update the list below when updating

Ah, so I bumped this here so that it didn't matter if we merged your PR that bumped in the mean time. I don't think we actually need to bump the schema version for this PR.

I think the list below still needs an summary of the sliding sync connection tables being added

oh yeah fair

synapse/storage/database.py

synapse/handlers/sliding_sync/store.py

synapse/storage/databases/main/sliding_sync.py

synapse/storage/schema/main/delta/87/02_per_connection_state.sql

synapse/storage/databases/main/sliding_sync.py

Co-authored-by: Eric Eastwood <[email protected]>

synapse/storage/databases/main/sliding_sync.py

Co-authored-by: Eric Eastwood <[email protected]>

MadLittleMods · 2024-08-27T18:32:06Z

synapse/handlers/sliding_sync/store.py

-        if sync_config.requester.access_token_id:
-            # If we don't have a device, then the access token ID should be a
-            # stable ID.
-            return (user_id, f"A/{sync_config.requester.access_token_id}/{conn_id}")


It looks like we're no longer taking access_token into account for the sliding_sync_connections table. Are we ok with the potential collisions?

If we care, we should add a test to make sure we don't regress this kind of thing.

Ugh, I'd forgotten I had a to do for that. My hunch is that ideally we'd always have sliding sync attached to a device so that eg we can tie it to last time we saw a device. We also don't generally use sliding sync with pupetted access (which is the common way we don't have a device). I'll have a think

I've renamed it to effective_device_id and added a comment.

Discussed in an internal meeting today

The column having the new name effective_device_id is good enough but we're not going to address taking access_token into account in this PR because the only users that won't have a device_id are application services. And since we block application services from using sync anyway, this doesn't matter now.

We can always support it in the future and the column already has a good name if we decide to support it.

synapse/storage/databases/main/sliding_sync.py

synapse/types/handlers/sliding_sync.py

This is so that we can cache it. We also move the sliding sync types to `synapse/types/handlers/sliding_sync.py`. This is mainly in-prep for #17599 to avoid circular imports. The only change in behaviour is that `RoomSyncConfig.combine_sync_config(..)` now returns a new room sync config rather than mutating in-place. Reviewable commit-by-commit. --------- Co-authored-by: Eric Eastwood <[email protected]>

This reverts commit ab414f2. Introduced in #17599

Follow-up to #17631 and #17632 to fix-up #17599 --------- Co-authored-by: Eric Eastwood <[email protected]>

Based on #17600 --------- Co-authored-by: Eric Eastwood <[email protected]>

@kfiven

# Synapse 1.114.0 (2024-09-02) This release enables support for [MSC4186](matrix-org/matrix-spec-proposals#4186) — Simplified Sliding Sync. This allows using the upcoming releases of the Element X mobile apps without having to run a Sliding Sync Proxy. ### Features - Enable native sliding sync support ([MSC3575](matrix-org/matrix-spec-proposals#3575) and [MSC4186](matrix-org/matrix-spec-proposals#4186)) by default. ([\#17648](element-hq/synapse#17648)) # Synapse 1.114.0rc3 (2024-08-30) ### Bugfixes - Fix regression in v1.114.0rc2 that caused workers to fail to start. ([\#17626](element-hq/synapse#17626)) # Synapse 1.114.0rc2 (2024-08-30) ### Features - Improve cross-signing upload when using [MSC3861](matrix-org/matrix-spec-proposals#3861) to use a custom UIA flow stage, with web fallback support. ([\#17509](element-hq/synapse#17509)) - Make `hash_password` script accept password input from stdin. ([\#17608](element-hq/synapse#17608)) ### Bugfixes - Fix hierarchy returning 403 when room is accessible through federation. Contributed by Krishan (@kfiven). ([\#17194](element-hq/synapse#17194)) - Fix content-length on federation `/thumbnail` responses. ([\#17532](element-hq/synapse#17532)) - Fix authenticated media responses using a wrong limit when following redirects over federation. ([\#17543](element-hq/synapse#17543)) ### Internal Changes - MSC3861: load the issuer and account management URLs from OIDC discovery. ([\#17407](element-hq/synapse#17407)) - Refactor sliding sync class into multiple files. ([\#17595](element-hq/synapse#17595)) - Store sliding sync per-connection state in the database. ([\#17599](element-hq/synapse#17599)) - Make the sliding sync `PerConnectionState` class immutable. ([\#17600](element-hq/synapse#17600)) - Add support to `@tag_args` for standalone functions. ([\#17604](element-hq/synapse#17604)) - Speed up incremental syncs in sliding sync by adding some more caching. ([\#17606](element-hq/synapse#17606)) - Always return the user's own read receipts in sliding sync. ([\#17617](element-hq/synapse#17617)) - Replace `isort` and `black` with `ruff`. ([\#17620](element-hq/synapse#17620)) - Refactor sliding sync code to move room list logic out into a separate class. ([\#17622](element-hq/synapse#17622)) ### Updates to locked dependencies * Bump attrs from 23.2.0 to 24.2.0. ([\#17609](element-hq/synapse#17609)) * Bump cryptography from 42.0.8 to 43.0.0. ([\#17584](element-hq/synapse#17584)) * Bump phonenumbers from 8.13.43 to 8.13.44. ([\#17610](element-hq/synapse#17610)) * Bump pygithub from 2.3.0 to 2.4.0. ([\#17612](element-hq/synapse#17612)) * Bump pyyaml from 6.0.1 to 6.0.2. ([\#17611](element-hq/synapse#17611)) * Bump sentry-sdk from 2.12.0 to 2.13.0. ([\#17585](element-hq/synapse#17585)) * Bump serde from 1.0.206 to 1.0.208. ([\#17581](element-hq/synapse#17581)) * Bump serde from 1.0.208 to 1.0.209. ([\#17613](element-hq/synapse#17613)) * Bump serde_json from 1.0.124 to 1.0.125. ([\#17582](element-hq/synapse#17582)) * Bump serde_json from 1.0.125 to 1.0.127. ([\#17614](element-hq/synapse#17614)) * Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813. ([\#17583](element-hq/synapse#17583)) * Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818. ([\#17586](element-hq/synapse#17586)) # Synapse 1.114.0rc1 (2024-08-20) ### Features - Add a flag to `/versions`, `org.matrix.simplified_msc3575`, to indicate whether experimental sliding sync support has been enabled. ([\#17571](element-hq/synapse#17571)) - Handle changes in `timeline_limit` in experimental sliding sync. ([\#17579](element-hq/synapse#17579)) - Correctly track read receipts that should be sent down in experimental sliding sync. ([\#17575](element-hq/synapse#17575), [\#17589](element-hq/synapse#17589), [\#17592](element-hq/synapse#17592)) ### Bugfixes - Start handlers for new media endpoints when media resource configured. ([\#17483](element-hq/synapse#17483)) - Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17510](element-hq/synapse#17510)) - Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately. ([\#17535](element-hq/synapse#17535)) - Better exclude partially stated rooms if we must await full state in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17538](element-hq/synapse#17538)) - Handle lower-case http headers in `_Mulitpart_Parser_Protocol`. ([\#17545](element-hq/synapse#17545)) - Fix fetching federation signing keys from servers that omit `old_verify_keys`. Contributed by @tulir @ Beeper. ([\#17568](element-hq/synapse#17568)) - Fix bug where we would respond with an error when a remote server asked for media that had a length of 0, using the new multipart federation media endpoint. ([\#17570](element-hq/synapse#17570)) ### Improved Documentation - Clarify default behaviour of the [`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites) option. ([\#17515](element-hq/synapse#17515)) - Improve docstrings for profile methods. ([\#17559](element-hq/synapse#17559)) ### Internal Changes - Add more tracing to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17514](element-hq/synapse#17514)) - Fixup comment in sliding sync implementation. ([\#17531](element-hq/synapse#17531)) - Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`. ([\#17536](element-hq/synapse#17536)) - Fix performance of device lists in `/key/changes` and sliding sync. ([\#17537](element-hq/synapse#17537), [\#17548](element-hq/synapse#17548)) - Bump setuptools from 67.6.0 to 72.1.0. ([\#17542](element-hq/synapse#17542)) - Add a utility function for generating random event IDs. ([\#17557](element-hq/synapse#17557)) - Speed up responding to media requests. ([\#17558](element-hq/synapse#17558), [\#17561](element-hq/synapse#17561), [\#17564](element-hq/synapse#17564), [\#17566](element-hq/synapse#17566), [\#17567](element-hq/synapse#17567), [\#17569](element-hq/synapse#17569)) - Test github token before running release script steps. ([\#17562](element-hq/synapse#17562)) - Reduce log spam of multipart files. ([\#17563](element-hq/synapse#17563)) - Refactor per-connection state in experimental sliding sync handler. ([\#17574](element-hq/synapse#17574)) - Add histogram metrics for sliding sync processing time. ([\#17593](element-hq/synapse#17593)) ### Updates to locked dependencies * Bump bytes from 1.6.1 to 1.7.1. ([\#17526](element-hq/synapse#17526)) * Bump lxml from 5.2.2 to 5.3.0. ([\#17550](element-hq/synapse#17550)) * Bump phonenumbers from 8.13.42 to 8.13.43. ([\#17551](element-hq/synapse#17551)) * Bump regex from 1.10.5 to 1.10.6. ([\#17527](element-hq/synapse#17527)) * Bump sentry-sdk from 2.10.0 to 2.12.0. ([\#17553](element-hq/synapse#17553)) * Bump serde from 1.0.204 to 1.0.206. ([\#17556](element-hq/synapse#17556)) * Bump serde_json from 1.0.122 to 1.0.124. ([\#17555](element-hq/synapse#17555)) * Bump sigstore/cosign-installer from 3.5.0 to 3.6.0. ([\#17549](element-hq/synapse#17549)) * Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808. ([\#17552](element-hq/synapse#17552)) * Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712. ([\#17524](element-hq/synapse#17524)) # Synapse 1.113.0 (2024-08-13) No significant changes since 1.113.0rc1. # Synapse 1.113.0rc1 (2024-08-06) ### Features - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) ### Bugfixes - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) ### Improved Documentation - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) ### Internal Changes - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) ### Updates to locked dependencies * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

erikjohnston added 2 commits August 22, 2024 17:04

Move sliding sync types

e2ade85

Update mypy plugin to handle enums and typevars

5b77f4a

erikjohnston mentioned this pull request Aug 22, 2024

Sliding Sync: Make PerConnectionState immutable #17600

Merged

erikjohnston changed the base branch from develop to erikj/ss_immutable August 22, 2024 16:25

erikjohnston force-pushed the erikj/ss_store_state branch 2 times, most recently from 341c500 to 6441b66 Compare August 22, 2024 16:48

erikjohnston force-pushed the erikj/ss_immutable branch from d1e5a21 to 9330abb Compare August 22, 2024 16:52

erikjohnston force-pushed the erikj/ss_store_state branch from 6441b66 to 15dbf71 Compare August 22, 2024 16:53

erikjohnston added 3 commits August 22, 2024 18:03

Make RoomSyncConfig immutable

7087c7c

Make PerConnectionState immutable

e34d634

Newsfile

87d5336

erikjohnston force-pushed the erikj/ss_immutable branch from 9330abb to 87d5336 Compare August 22, 2024 17:04

erikjohnston added 5 commits August 22, 2024 18:04

Allow making columns AUTOINCREMENT

d1ee253

Add simple_insert_returning_txn

b3d8e2d

Store state

3838b18

Remove mark_token_seen

ed7591c

Newsfile

03eac5a

erikjohnston force-pushed the erikj/ss_store_state branch from 15dbf71 to 03eac5a Compare August 22, 2024 17:04

erikjohnston marked this pull request as ready for review August 22, 2024 17:12

erikjohnston requested a review from a team as a code owner August 22, 2024 17:12

erikjohnston added the A-Sync label Aug 22, 2024

erikjohnston requested a review from MadLittleMods August 22, 2024 17:12

MadLittleMods reviewed Aug 26, 2024

View reviewed changes

erikjohnston and others added 6 commits August 26, 2024 20:02

Apply suggestions from code review

7935423

Co-authored-by: Eric Eastwood <[email protected]>

Add comment about why we ignore zero position

948456b

Don't bother to use zip

68a2a98

Add timestamp index

8ed1c07

Rename column

bae50d3

Fix errors

1e5a3a7

MadLittleMods reviewed Aug 27, 2024

View reviewed changes

synapse/storage/databases/main/sliding_sync.py Outdated Show resolved Hide resolved

erikjohnston and others added 4 commits August 27, 2024 11:55

Rename get_and_clear_connection_positions

0e07f65

Don't assert unkown streams, log

4a68975

Index on created ts

ac14e57

Update synapse/types/handlers/sliding_sync.py

5fe6466

Co-authored-by: Eric Eastwood <[email protected]>

erikjohnston requested a review from MadLittleMods August 27, 2024 12:04

MadLittleMods reviewed Aug 27, 2024

View reviewed changes

synapse/storage/databases/main/sliding_sync.py Show resolved Hide resolved

MadLittleMods reviewed Aug 27, 2024

View reviewed changes

synapse/storage/databases/main/sliding_sync.py Show resolved Hide resolved

MadLittleMods reviewed Aug 27, 2024

View reviewed changes

synapse/types/handlers/sliding_sync.py Show resolved Hide resolved

erikjohnston requested a review from MadLittleMods August 28, 2024 09:10

erikjohnston added 4 commits August 28, 2024 17:43

Add cast comment

9065382

Rename to 'effective_device_id'

52f2199

Merge remote-tracking branch 'origin/develop' into erikj/ss_immutable

174e1ad

Merge branch 'erikj/ss_immutable' into erikj/ss_store_state

5152f19

MadLittleMods approved these changes Aug 29, 2024

View reviewed changes

erikjohnston added 2 commits August 29, 2024 16:17

Merge remote-tracking branch 'origin/develop' into erikj/ss_store_state

8a54f48

Add commente on new tables in in schema notes

2da80fe

github-actions bot deployed to PR Documentation Preview August 29, 2024 15:21 Active

Base automatically changed from erikj/ss_immutable to develop August 29, 2024 15:23

Merge remote-tracking branch 'origin/develop' into erikj/ss_store_state

57a9506

erikjohnston merged commit e43c2b0 into develop Aug 29, 2024
29 checks passed

erikjohnston deleted the erikj/ss_store_state branch August 29, 2024 15:27

erikjohnston mentioned this pull request Aug 29, 2024

Fix background update for sliding sync (find previous membership) (v1) #17631

Merged

erikjohnston added a commit that referenced this pull request Aug 29, 2024

Fix background update for sliding sync (#17631)

bb80894

This reverts commit ab414f2. Introduced in #17599

erikjohnston mentioned this pull request Aug 29, 2024

Fix background update for sliding sync (find previous membership) (v2) #17632

Merged

MadLittleMods mentioned this pull request Aug 29, 2024

Sliding sync: Fix bg update again (v3) #17634

Merged

erikjohnston added a commit that referenced this pull request Aug 30, 2024

Sliding sync: Fix bg update again (v3) (#17634)

7098d47

Follow-up to #17631 and #17632 to fix-up #17599 --------- Co-authored-by: Eric Eastwood <[email protected]>

erikjohnston added a commit that referenced this pull request Aug 30, 2024

Sliding sync: Store the per-connection state in the database. (#17599)

b913aaa

Based on #17600 --------- Co-authored-by: Eric Eastwood <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sliding sync: Store the per-connection state in the database. #17599

Sliding sync: Store the per-connection state in the database. #17599

erikjohnston commented Aug 22, 2024 •

edited

Loading

MadLittleMods Aug 26, 2024

erikjohnston Aug 26, 2024

MadLittleMods Aug 29, 2024

erikjohnston Aug 29, 2024

erikjohnston Aug 29, 2024

MadLittleMods Aug 27, 2024 •

edited

Loading

erikjohnston Aug 27, 2024

erikjohnston Aug 28, 2024

MadLittleMods Aug 29, 2024 •

edited

Loading

Sliding sync: Store the per-connection state in the database. #17599

Sliding sync: Store the per-connection state in the database. #17599

Conversation

erikjohnston commented Aug 22, 2024 • edited Loading

MadLittleMods Aug 26, 2024

Choose a reason for hiding this comment

erikjohnston Aug 26, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 29, 2024

Choose a reason for hiding this comment

erikjohnston Aug 29, 2024

Choose a reason for hiding this comment

erikjohnston Aug 29, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

erikjohnston Aug 27, 2024

Choose a reason for hiding this comment

erikjohnston Aug 28, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 29, 2024 • edited Loading

Choose a reason for hiding this comment

erikjohnston commented Aug 22, 2024 •

edited

Loading

MadLittleMods Aug 27, 2024 •

edited

Loading

MadLittleMods Aug 29, 2024 •

edited

Loading