Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748

MadLittleMods · 2023-06-08T22:50:25Z

Remove experimental MSC2716 implementation to incrementally import history into existing rooms

Context for why we're removing the implementation:

MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)
MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)

Spurred on by wanting to optimize get_backfill_events(...) as part of #15653 without dealing with the MSC2716 logic.

Anyone wanting to continue MSC2716, should also address these leftover tasks: #10737

Closes #10737 in the fact that it is not longer necessary to track those things.

Todo

Remove database tables -> Gracefully drop MSC2716 related tables (incrementally import history) #15786
- insertion_events
- insertion_event_edges
- insertion_event_extremities
- batch_events
- See Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748 (comment)
Potentially remove ... argument threading through to everything (like state_event_ids and allow_no_prev_events)
- We can handle this in future PRs as we see fit
Remove event.internal_metadata.is_historical and m.historical things

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

Conflicts: tests/storage/test_event_federation.py

MadLittleMods · 2023-06-09T20:54:43Z

changelog.d/15748.removal

@@ -0,0 +1 @@
+Remove experimental [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716) implementation to incrementally import history into existing rooms.


Do we need to worry about gracefully dropping database tables for an experimental feature like this?

Relevant tables to drop:

-- Drop tables related to MSC2716 since the implementation is being removed DROP TABLE insertion_events; DROP TABLE insertion_event_edges; DROP TABLE insertion_event_extremities; DROP TABLE batch_events;

Can we just stop writing to the tables in the Synapse version that this PR ships in, then in the next Synapse version, bump the SCHEMA_COMPAT_VERSION and drop the tables?

I'd say we may as well? We still want to be careful that we don't accidentally break upgrades/downgrades for people that have turned on experimental features.

Easiest way of tracking the task of dropping those tables is to create an issue with the future maintenance label and add it to one of the milestones.

Created #15786 to track this ⏩

erikjohnston · 2023-06-13T09:19:16Z

synapse/api/room_versions.py

        msc3389_relation_redactions=False,
        msc3787_knock_restricted_join_rule=True,
        msc3667_int_only_power_levels=True,
        msc3821_redaction_rules=False,
        msc3931_push_features=(),
        msc3989_redaction_rules=False,
    )
-    MSC2716v4 = RoomVersion(
-        "org.matrix.msc2716v4",


Removing this room version will mean that users won't be able to leave / interact with such rooms. Not sure if that's a problem for this feature or not

ehh, people are going to get left behind regardless of the time period we give and I don't think this was very widely used. I've created #15785 to track a solution to this problem in general.

erikjohnston

Otherwise I think this looks good

changelog.d/15748.removal

MadLittleMods · 2023-06-16T19:13:10Z

Thanks for the review @erikjohnston 🪹

Context for why we're removing the implementation: - matrix-org/matrix-spec-proposals#2716 (comment) - matrix-org/matrix-spec-proposals#2716 (comment) Removed from Synapse in matrix-org/synapse#15748

@PaarthShah

Please note that this will be the last release of Synapse that is compatible with Python 3.7 and earlier. This is due to Python 3.7 now having reached End of Life; see our [deprecation policy](https://matrix-org.github.io/synapse/v1.87/deprecation_policy.html) for more details. - Pin `pydantic` to `^1.7.4` to avoid backwards-incompatible API changes from the 2.0.0 release. Resolves matrix-org#15858. Contributed by @PaarthShah. ([\matrix-org#15862](matrix-org#15862)) - Split out 2022 changes from the changelog so the rendered version in GitHub doesn't timeout as much. ([\matrix-org#15846](matrix-org#15846)) - Improve `/messages` response time by avoiding backfill when we already have messages to return. ([\matrix-org#15737](matrix-org#15737)) - Add spam checker module API for logins. ([\matrix-org#15838](matrix-org#15838)) - Fix a long-standing bug where media files were served in an unsafe manner. Contributed by @joshqou. ([\matrix-org#15680](matrix-org#15680)) - Avoid invalidating a cache that was just prefilled. ([\matrix-org#15758](matrix-org#15758)) - Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15770](matrix-org#15770)) - Fix joining rooms through aliases where the alias server isn't a real homeserver. Contributed by @tulir @ Beeper. ([\matrix-org#15776](matrix-org#15776)) - Fix a bug in push rules handling leading to an invalid (per spec) `is_user_mention` rule sent to clients. Also fix wrong rule names for `is_user_mention` and `is_room_mention`. ([\matrix-org#15781](matrix-org#15781)) - Fix a bug introduced in 1.57.0 where the wrong table would be locked on updating database rows when using SQLite as the database backend. ([\matrix-org#15788](matrix-org#15788)) - Fix Sytest environmental variable evaluation in CI. ([\matrix-org#15804](matrix-org#15804)) - Fix forgotten rooms missing from initial sync after rejoining them. Contributed by Nico from Famedly. ([\matrix-org#15815](matrix-org#15815)) - Fix sqlite `user_filters` upgrade introduced in v1.86.0. ([\matrix-org#15817](matrix-org#15817)) - Document `looping_call()` functionality that will wait for the given function to finish before scheduling another. ([\matrix-org#15772](matrix-org#15772)) - Fix a typo in the [Admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html). ([\matrix-org#15805](matrix-org#15805)) - Fix typo in MSC number in faster remote room join architecture doc. ([\matrix-org#15812](matrix-org#15812)) - Remove experimental [MSC2716](matrix-org/matrix-spec-proposals#2716) implementation to incrementally import history into existing rooms. ([\matrix-org#15748](matrix-org#15748)) - Replace `EventContext` fields `prev_group` and `delta_ids` with field `state_group_deltas`. ([\matrix-org#15233](matrix-org#15233)) - Regularly try to send transactions to other servers after they failed instead of waiting for a new event to be available before trying. ([\matrix-org#15743](matrix-org#15743)) - Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15755](matrix-org#15755)) - Allow for the configuration of max request retries and min/max retry delays in the matrix federation client. ([\matrix-org#15783](matrix-org#15783)) - Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic. ([\matrix-org#15806](matrix-org#15806)) - Fix harmless exceptions being printed when running the port DB script. ([\matrix-org#15814](matrix-org#15814)) * Bump attrs from 22.2.0 to 23.1.0. ([\matrix-org#15801](matrix-org#15801)) * Bump cryptography from 40.0.2 to 41.0.1. ([\matrix-org#15800](matrix-org#15800)) * Bump ijson from 3.2.0.post0 to 3.2.1. ([\matrix-org#15802](matrix-org#15802)) * Bump phonenumbers from 8.13.13 to 8.13.14. ([\matrix-org#15798](matrix-org#15798)) * Bump ruff from 0.0.265 to 0.0.272. ([\matrix-org#15799](matrix-org#15799)) * Bump ruff from 0.0.272 to 0.0.275. ([\matrix-org#15833](matrix-org#15833)) * Bump serde_json from 1.0.96 to 1.0.97. ([\matrix-org#15797](matrix-org#15797)) * Bump serde_json from 1.0.97 to 1.0.99. ([\matrix-org#15832](matrix-org#15832)) * Bump towncrier from 22.12.0 to 23.6.0. ([\matrix-org#15831](matrix-org#15831)) * Bump types-opentracing from 2.4.10.4 to 2.4.10.5. ([\matrix-org#15830](matrix-org#15830)) * Bump types-setuptools from 67.8.0.0 to 68.0.0.0. ([\matrix-org#15835](matrix-org#15835))

MadLittleMods added 4 commits May 24, 2023 13:54

Remove MSC2716 /batch_send endpoint

be3c21e

WIP: Remove surrounding things

f0528e7

Merge branch 'develop' into madlittlemods/remove-msc2716

f3243b0

Conflicts: tests/storage/test_event_federation.py

Remove more references

2778724

MadLittleMods added the T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. label Jun 8, 2023

MadLittleMods added 6 commits June 8, 2023 18:52

Remove stray batch_send reference

4845799

Remove MSC2716 being enabled in Complement tests

4652004

Remove historical distinction on events

3779b04

Remove leftover _handle_marker_event usage

35198b1

Only one type of backfill point now

046e6d2

Remove leftover MSC2716 storage related things

52864ce

github-actions bot deployed to PR Documentation Preview June 9, 2023 00:17 View deployment

Add changelog

d8a30e3

github-actions bot deployed to PR Documentation Preview June 9, 2023 20:52 View deployment

MadLittleMods commented Jun 9, 2023

View reviewed changes

MadLittleMods marked this pull request as ready for review June 10, 2023 02:01

MadLittleMods requested a review from a team as a code owner June 10, 2023 02:01

erikjohnston reviewed Jun 13, 2023

View reviewed changes

Merge branch 'develop' into madlittlemods/remove-msc2716

d497859

github-actions bot deployed to PR Documentation Preview June 15, 2023 21:24 View deployment

MadLittleMods mentioned this pull request Jun 15, 2023

Gracefully drop MSC2716 related tables (incrementally import history) #15786

Closed

MadLittleMods commented Jun 15, 2023

View reviewed changes

changelog.d/15748.removal Show resolved Hide resolved

MadLittleMods requested a review from erikjohnston June 15, 2023 21:56

erikjohnston approved these changes Jun 16, 2023

View reviewed changes

Merge branch 'develop' into madlittlemods/remove-msc2716

dbc0632

github-actions bot deployed to PR Documentation Preview June 16, 2023 17:41 View deployment

MadLittleMods merged commit 0f02f0b into develop Jun 16, 2023

MadLittleMods deleted the madlittlemods/remove-msc2716 branch June 16, 2023 19:12

MadLittleMods mentioned this pull request Jun 16, 2023

Remove MSC2716 tests to incrementally import history into existing rooms matrix-org/complement#640

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748

Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748

MadLittleMods commented Jun 8, 2023 •

edited

Loading

MadLittleMods Jun 9, 2023 •

edited

Loading

erikjohnston Jun 13, 2023

MadLittleMods Jun 15, 2023

erikjohnston Jun 13, 2023

MadLittleMods Jun 15, 2023

erikjohnston left a comment

MadLittleMods commented Jun 16, 2023

		@@ -0,0 +1 @@
		Remove experimental [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716) implementation to incrementally import history into existing rooms.

Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748

Remove experimental MSC2716 implementation to incrementally import history into existing rooms #15748

Conversation

MadLittleMods commented Jun 8, 2023 • edited Loading

Todo

Pull Request Checklist

MadLittleMods Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

erikjohnston Jun 13, 2023

Choose a reason for hiding this comment

MadLittleMods Jun 15, 2023

Choose a reason for hiding this comment

erikjohnston Jun 13, 2023

Choose a reason for hiding this comment

MadLittleMods Jun 15, 2023

Choose a reason for hiding this comment

erikjohnston left a comment

Choose a reason for hiding this comment

MadLittleMods commented Jun 16, 2023

MadLittleMods commented Jun 8, 2023 •

edited

Loading

MadLittleMods Jun 9, 2023 •

edited

Loading