Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add a Synapse Module for configuring presence update routing #9491

Merged
merged 35 commits into from
Apr 6, 2021

Conversation

anoadragon453
Copy link
Member

@anoadragon453 anoadragon453 commented Feb 24, 2021

Based on #9650

At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, get_users_for_states and get_interested_users. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to ModuleApi which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

  • Sending state for a specific set or all known users to a defined set of local and remote users.
  • The ability to trigger an initial sync for specific users, so they receive all current state.

This is quite a chunky PR, though hopefully the commits should have enough explanation in each of them to get you by.

@anoadragon453 anoadragon453 added the Z-Time-Tracked Element employees should track their time spent on this issue/PR. label Feb 24, 2021
@erikjohnston
Copy link
Member

  • The ability to trigger an initial sync for specific users, so they receive all current state. (TODO)

What do you mean by this?

Something to consider: we may want to do some validation on Synapse's side to ensure modules aren't generating their own state? Or is that a feature, rather than a bug.

I don't think its worth trying to enforce TBH,

@erikjohnston
Copy link
Member

I've also realised that that there is another place where we have to hook in, which is where /sync tries to get all other users' presence the syncing user is interested in:

@cached(num_args=2, cache_context=True)
async def _get_interested_in(self, user, explicit_room_id, cache_context):
"""Returns the set of users that the given user should see presence
updates for
"""
user_id = user.to_string()
users_interested_in = set()
users_interested_in.add(user_id) # So that we receive our own presence
users_who_share_room = await self.store.get_users_who_share_room_with_user(
user_id, on_invalidate=cache_context.invalidate
)
users_interested_in.update(users_who_share_room)
if explicit_room_id:
user_ids = await self.store.get_users_in_room(
explicit_room_id, on_invalidate=cache_context.invalidate
)
users_interested_in.update(user_ids)
return users_interested_in

@anoadragon453
Copy link
Member Author

  • The ability to trigger an initial sync for specific users, so they receive all current state. (TODO)

What do you mean by this?

When another user is determined to receive presence for all users, it's crucial to be able to send them all existing presence of all local users - not just the latest updates as the proposal currently provides. Thus, either:

  1. Synapse, upon detecting a change from what the module returned last, could send all current user presence to the new user. This would ensure they are always kept up to date. However, would be wildly inefficient if the module were changing the list of returned users constantly, plus I think it reduces the flexibility of the module.
  2. get_rooms_and_users_for_states could return another list containing users that should receive all local user presence. That still sounds a bit finicky though, especially if we're updating the list of users to send presence to asynchronously.
  3. We could add a new method to ModuleApi that allows modules to, at any point, send all local user presence to a given set of users. This would allow the module's logic to determine when this should happen, separately from when get_rooms_and_users_for_states is called.

I'm leaning towards 3. My only reservation is whether this would be useful in the generic sense but... probably?

Something to consider: we may want to do some validation on Synapse's side to ensure modules aren't generating their own state? Or is that a feature, rather than a bug.

I don't think its worth trying to enforce TBH,

Yeah, probably not. The sysadmin did decide to install that module on their own 🙂

@anoadragon453
Copy link
Member Author

@erikjohnston I've gone with 3. for now and defined ModuleApi.send_local_online_presence_to(users: List[str]) -> None, which can be used to update a user on the current state of local user presence, which is useful if they've been offline for a while.

@anoadragon453
Copy link
Member Author

For the implementation of ModuleApi.send_local_online_presence_to(users: List[str]) -> None, I'm envisioning an implementation along the lines of the following:

self._presence_stream = self.hs.get_event_sources().sources["presence"]
#...
def ModuleApi.send_local_online_presence_to(users: List[str]) -> None:
    for user in users:
        if self.hs.is_mine(user):
            # Modify SyncHandler._generate_sync_entry_for_presence to call
            # presence_source.get_new_events with an empty `from_key` if
            # that user's ID were in a list modified by ModuleApi somewhere.
            # That user would then get all presence state on next incremental sync.
        else:
            # Retrieve presence state for all currently online users
            presence_events = await self._presence_stream.get_new_events(
                 user, from_key=None, include_offline=False
            )
            FederationSender.send_presence([ev.state for ev in presence_events])

Technically this method is only needed for remote users (local users could just initial sync), but it's much more convenient if the module can specify both.

@erikjohnston erikjohnston self-requested a review March 3, 2021 12:17
@erikjohnston
Copy link
Member

Basically yeah I think that works. Instead of broadcasting all local presence states it should only send states that the given user is interested in though.

@anoadragon453
Copy link
Member Author

Basically yeah I think that works. Instead of broadcasting all local presence states it should only send states that the given user is interested in though.

Oh true. And that could be every local user if used in conjunction with get_rooms_and_users_for_states.

Comment on lines 87 to 98
async def get_users_for_states(
self,
state_updates: Iterable[UserPresenceState],
) -> Dict[str, Set[UserPresenceState]]:
"""Given an iterable of user presence updates, determine where each one
needs to go.

Args:
state_updates: An iterable of user presence state updates.

Returns:
A dictionary of user_id -> set of UserPresenceState that the user should
receive.
"""
destination_users = {} # type: Dict[str, Set[UserPresenceState]

# Ignore any updates for blacklisted users
desired_updates = set()
for update in state_updates:
if update.state_key not in self._config.blacklisted_users:
desired_updates.add(update)

# Send all presence updates to specific users
for user_id in self._config.always_send_to_users:
destination_users[user_id] = desired_updates

return destination_users
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this method works great for when you have a set of state updates you want to send out. But I'm increasingly realising it works terribly for when you have a user that you want to query the current state of other users for.

Where we were looking to hook in, we'd need to somehow return user IDs of users on the server that we should query presence for. get_users_for_states expects a set of UserPresenceState which at this point we don't have yet. We actually purposefully don't have that yet. At the linked call, we try to just figure out the IDs of users we want to pull presence of, instead of pulling all the presence right away. If we do that (so that we can pass UserPresenceState instances to get_users_for_states), then we lose all of the optimisation this function is aiming for (only pulling state at the very end after we've filtered out users).

We also can't just add another method that says "for a given user syncing, which users should we try to get updates for?" as the module won't know the user IDs of anyone else on the homeserver except those it's hoping to route presence to.

So I'm a bit stuck on how to support the collection of presence through /sync while honoring what get_users_for_states returns at the moment.

Copy link
Member Author

@anoadragon453 anoadragon453 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I meant to link the following for where we planned to hook into the /sync code:

users_interested_in = await self._get_interested_in(user, explicit_room_id)

Copy link
Member

@erikjohnston erikjohnston Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Looking at the sync code I wonder if we can make _get_interested_in return a "all" value, which we can then filter after pulling out the states that have changed:

users_interested_in = await self._get_interested_in(user, explicit_room_id)
user_ids_changed = set() # type: Collection[str]
changed = None
if from_key:
changed = stream_change_cache.get_all_entities_changed(from_key)
if changed is not None and len(changed) < 500:
assert isinstance(user_ids_changed, set)
# For small deltas, its quicker to get all changes and then
# work out if we share a room or they're in our presence list
get_updates_counter.labels("stream").inc()
for other_user_id in changed:
if other_user_id in users_interested_in:
user_ids_changed.add(other_user_id)
else:
# Too many possible updates. Find all users we can see and check
# if any of them have changed.
get_updates_counter.labels("full").inc()
if from_key:
user_ids_changed = stream_change_cache.get_entities_changed(
users_interested_in, from_key
)
else:
user_ids_changed = users_interested_in
updates = await presence.current_state_for_users(user_ids_changed)

I think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, however get_users_for_states currently supports routing a subset of incoming presence to some users, rather than all of the presence.

So while that would support the "all" usecase, it wouldn't work if the module wanted to only route some presence to a user.

We could just throw our hands up and make get_users_for_states only support routing all local user presence, but the generality is further diminished.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that we could have a function get_interested_in(user) -> Union[Set[str], All], to figure out which presence updates to fetch, and then pass those to another function to filter them down correctly (or reuse get_users_for_states).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, so that proposed function would be in the module, correct? That sounds doable.

My only concern is that if get_interested_in is not implemented by the module, then get_users_for_states may not work as expected for local users. When local presence updates come in, get_users_for_states is simply used to figure out which user /sync streams to wake up. We would then query get_interested_in to determine which state events to pull.

So we'd need to trust the module developer to return similar things for both functions. We is fine and we could document that, though ideally we'd just have a single function that could be called for both cases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. Though we can add strong "here be dragons" warnings.

I wonder if we can write default implementations, get_interested_in always returning all and then use get_users_for_states as the way of filtering?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would work... and then it would just be a performance hit if the module didn't implement get_interested_in. And worst case is the desyncronisation of results, but hopefully the documentation would do enough here to make not doing that clear.

I'll move forward with that for now, thanks!

@anoadragon453 anoadragon453 marked this pull request as ready for review March 15, 2021 17:54
konradkonrad added a commit to konradkonrad/raiden-service-bundle that referenced this pull request Mar 17, 2021
Note: this builds on top of a pull request branch by https://github.com/anoa

When matrix-org/synapse#9491 is released, switch back to installing SYNAPSE from pypi.
@anoadragon453 anoadragon453 force-pushed the anoa/presence_hook branch 4 times, most recently from 991842d to b588f16 Compare March 19, 2021 11:59
@anoadragon453 anoadragon453 requested a review from a team March 19, 2021 12:22
docs/presence_router_module.md Show resolved Hide resolved
docs/presence_router_module.md Show resolved Hide resolved
synapse/events/presence_router.py Outdated Show resolved Hide resolved
synapse/module_api/__init__.py Outdated Show resolved Hide resolved
synapse/handlers/presence.py Outdated Show resolved Hide resolved
@@ -0,0 +1,222 @@
# Presence Router Module
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was asked today about how one would implement this module for an application service, instead of a user calling /sync, and had a bit of a look through the code. I noticed a couple of things:

  1. There's some code that determines whether an application service is interested in a user or a room. This is used to check whether we need to consider sending an event linked to a room/user to the application service. We don't hook into this yet. I suppose we could add a call to get_interested_users here and call it with the user_id argument set to the Application Service's sender_localpart user? get_users_for_states would still be called to filter presence updates once an application service had been declared interested.

    This could also be worked around by having the AS expand their namespace regexes, thus making them interested in a wide range of users (potentially all), but I think that's a bit hacky.

  2. The ModuleApi.send_local_online_presence_to method is currently only set up to modify /sync responses. I know we're trying to discourage AS users from syncing. So we probably need to modify the method to check if the user is a known application service sender_localpart, and if so send a burst of all current user presence states in a transaction.

I'll also add some tests for application services to verify what does and doesn't already work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After testing locally appservices do not currently receive presence updates.

However in the interest of not complicating this PR further, I'm going to leave a note about this and aim to fix it in a future PR.

anoadragon453 added a commit that referenced this pull request Mar 25, 2021
#9650)

Split off from #9491

Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133

Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
This class will perform in the same manner as Synapse did before,
unless a custom PresenceRouter module is configured. If one is,
then it will pass through the calls from Synapse to that module.
@anoadragon453
Copy link
Member Author

So tests broke briefly because we were attempting to load a FederationSender in ModuleApi's __init__, which some tests call without having set send_federation: True in their config (as expected).

So I now only load it in send_local_online_presence_to and added notes everywhere cautioning that the method must only be called on workers that support federation sending.

That lead me to another thought though, in that if you configure a PresenceRouter per worker, and your module is doing things like running ModuleApi.send_local_online_presence_to when something changes externally, that's going to happen multiple times over. So modules probably need an option to prevent more than one instance trying to broadcast presence information.

Any idea where I could add a note for that, or whether there's any precedent for configuring modules per-worker thus far? (I imagine you'd just stick a presence.presence_router block in each worker config...)

@erikjohnston
Copy link
Member

That lead me to another thought though, in that if you configure a PresenceRouter per worker, and your module is doing things like running ModuleApi.send_local_online_presence_to when something changes externally, that's going to happen multiple times over. So modules probably need an option to prevent more than one instance trying to broadcast presence information.

Hmm, I think its fine so long as send_local_online_presence_to is only called in response to the presence handler poking it, as that will then only get run on one worker? This hasn't been a problem for any modules before as they tend to only do stuff when poked by Synapse

synapse/handlers/presence.py Outdated Show resolved Hide resolved
docs/presence_router_module.md Outdated Show resolved Hide resolved
docs/presence_router_module.md Outdated Show resolved Hide resolved
docs/presence_router_module.md Outdated Show resolved Hide resolved
docs/presence_router_module.md Outdated Show resolved Hide resolved

updates = await presence.current_state_for_users(user_ids_changed)
return list(users_to_state.values()), max_token
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we not decide that we wanted to filter users through get_users_for_states if we returned ALL?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We did, and this is indeed a path that does not do so! I've made this path use _filter_all_presence_updates_for_user in 2f16ed0, as well as made that function a bit cleaner.

@anoadragon453
Copy link
Member Author

Hmm, I think its fine so long as send_local_online_presence_to is only called in response to the presence handler poking it, as that will then only get run on one worker?

Right, however send_local_online_presence_to was specifically designed to be called when the list of presence-receiving users was appended to. That can happen in response to something changing externally (say if an external, block-like data structure updated to include more users 😉).

I think this is fine, and the module implementer would just need to be aware and only call the module on the right workers.

This commit refactors _filter_all_presence_updates_for_user a little bit to allow
taking from_key as an optional parameter. The advantage of this function is that
we filter presence updates through PresenceRouter. Turns out we had a code path
(no from_key, return presence for ALL users) that ended up not filtering through
PresenceRouter.

Having both code paths end in _filter_all_presence_updates_for_user not only makes
things a little easier to follow, but it ensures filtering always happens.

I also took the bit where we remove the user from
ModuleApi.send_full_presence_to_local_users out of
_filter_all_presence_updates_for_user to clean it up a bit further.
This test was more concerned with the functionality of PresenceRouter, so I moved it
to the PresenceRouter test file instead.
The current default_config method didn't quite work for the test we just moved in,
(it negated any use of @override_config). So convert it to an @override_config
call instead.
@erikjohnston
Copy link
Member

@anoadragon453 the isort is failing :)

@anoadragon453
Copy link
Member Author

🤦 fixed.

@anoadragon453 anoadragon453 merged commit 0481923 into develop Apr 6, 2021
@anoadragon453 anoadragon453 deleted the anoa/presence_hook branch April 6, 2021 13:38
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 28, 2021
Synapse 1.32.2 (2021-04-22)
===========================

This release includes a fix for a regression introduced in 1.32.0.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 and 1.32.1 which caused `LoggingContext` errors in plugins. ([\#9857](matrix-org/synapse#9857))


Synapse 1.32.1 (2021-04-21)
===========================

This release fixes [a regression](matrix-org/synapse#9853)
in Synapse 1.32.0 that caused connected Prometheus instances to become unstable.

However, as this release is still subject to the `LoggingContext` change in 1.32.0,
it is recommended to remain on or downgrade to 1.31.0.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 which caused Synapse to report large numbers of Prometheus time series, potentially overwhelming Prometheus instances. ([\#9854](matrix-org/synapse#9854))


Synapse 1.32.0 (2021-04-20)
===========================

**Note:** This release introduces [a regression](matrix-org/synapse#9853)
that can overwhelm connected Prometheus instances. This issue was not present in
1.32.0rc1. If affected, it is recommended to downgrade to 1.31.0 in the meantime, and
follow [these instructions](matrix-org/synapse#9854 (comment))
to clean up any excess writeahead logs.

**Note:** This release also mistakenly included a change that may affected Synapse
modules that import `synapse.logging.context.LoggingContext`, such as
[synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider).
This will be fixed in a later Synapse version.

**Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+.

This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities.

This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date.

If you are using the `packages.matrix.org` Debian repository for Synapse packages,
note that we have recently updated the expiry date on the gpg signing key. If you see an
error similar to `The following signatures were invalid: EXPKEYSIG F473DD4473365DE1`, you
will need to get a fresh copy of the keys. You can do so with:

```sh
sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
```

Bugfixes
--------

- Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](matrix-org/synapse#9829))


Synapse 1.32.0rc1 (2021-04-13)
==============================

Features
--------

- Add a Synapse module for routing presence updates between users. ([\#9491](matrix-org/synapse#9491))
- Add an admin API to manage ratelimit for a specific user. ([\#9648](matrix-org/synapse#9648))
- Include request information in structured logging output. ([\#9654](matrix-org/synapse#9654))
- Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](matrix-org/synapse#9691))
- Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](matrix-org/synapse#9700))
- Add experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9717](matrix-org/synapse#9717), [\#9735](matrix-org/synapse#9735))
- Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](matrix-org/synapse#9710))
- Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](matrix-org/synapse#9766))


Bugfixes
--------

- Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](matrix-org/synapse#8926))
- Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](matrix-org/synapse#9711))
- Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](matrix-org/synapse#9725))
- Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](matrix-org/synapse#9770))
- Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](matrix-org/synapse#9780))


Updates to the Docker image
---------------------------

- Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](matrix-org/synapse#9765))


Improved Documentation
----------------------

- Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](matrix-org/synapse#9719))


Deprecations and Removals
-------------------------

- Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](matrix-org/synapse#9401))
- Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](matrix-org/synapse#9548))


Internal Changes
----------------

- Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](matrix-org/synapse#9718))
- Experiment with GitHub Actions for CI. ([\#9661](matrix-org/synapse#9661))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](matrix-org/synapse#9682))
- Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](matrix-org/synapse#9685))
- Improve Jaeger tracing for `to_device` messages. ([\#9686](matrix-org/synapse#9686))
- Add release helper script for automating part of the Synapse release process. ([\#9713](matrix-org/synapse#9713))
- Add type hints to expiring cache. ([\#9730](matrix-org/synapse#9730))
- Convert various testcases to `HomeserverTestCase`. ([\#9736](matrix-org/synapse#9736))
- Start linting mypy with `no_implicit_optional`. ([\#9742](matrix-org/synapse#9742))
- Add missing type hints to federation handler and server. ([\#9743](matrix-org/synapse#9743))
- Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](matrix-org/synapse#9753))
- Fix incompatibility with `tox` 2.5. ([\#9769](matrix-org/synapse#9769))
- Enable Complement tests for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary API. ([\#9771](matrix-org/synapse#9771))
- Use mock from the standard library instead of a separate package. ([\#9772](matrix-org/synapse#9772))
- Update Black configuration to target Python 3.6. ([\#9781](matrix-org/synapse#9781))
- Add option to skip unit tests when building Debian packages. ([\#9793](matrix-org/synapse#9793))


Synapse 1.31.0 (2021-04-06)
===========================

**Note:** As announced in v1.25.0, and in line with the deprecation policy for platform dependencies, this is the last release to support Python 3.5 and PostgreSQL 9.5. Future versions of Synapse will require Python 3.6+ and PostgreSQL 9.6+, as per our [deprecation policy](docs/deprecation_policy.md).

This is also the last release that the Synapse team will be publishing packages for Debian Stretch and Ubuntu Xenial.


Improved Documentation
----------------------

- Add a document describing the deprecation policy for platform dependencies. ([\#9723](matrix-org/synapse#9723))


Internal Changes
----------------

- Revert using `dmypy run` in lint script. ([\#9720](matrix-org/synapse#9720))
- Pin flake8-bugbear's version. ([\#9734](matrix-org/synapse#9734))


Synapse 1.31.0rc1 (2021-03-30)
==============================

Features
--------

- Add support to OpenID Connect login for requiring attributes on the `userinfo` response. Contributed by Hubbe King. ([\#9609](matrix-org/synapse#9609))
- Add initial experimental support for a "space summary" API. ([\#9643](matrix-org/synapse#9643), [\#9652](matrix-org/synapse#9652), [\#9653](matrix-org/synapse#9653))
- Add support for the busy presence state as described in [MSC3026](matrix-org/matrix-spec-proposals#3026). ([\#9644](matrix-org/synapse#9644))
- Add support for credentials for proxy authentication in the `HTTPS_PROXY` environment variable. ([\#9657](matrix-org/synapse#9657))


Bugfixes
--------

- Fix a longstanding bug that could cause issues when editing a reply to a message. ([\#9585](matrix-org/synapse#9585))
- Fix the `/capabilities` endpoint to return `m.change_password` as disabled if the local password database is not used for authentication. Contributed by @dklimpel. ([\#9588](matrix-org/synapse#9588))
- Check if local passwords are enabled before setting them for the user. ([\#9636](matrix-org/synapse#9636))
- Fix a bug where federation sending can stall due to `concurrent access` database exceptions when it falls behind. ([\#9639](matrix-org/synapse#9639))
- Fix a bug introduced in Synapse 1.30.1 which meant the suggested `pip` incantation to install an updated `cryptography` was incorrect. ([\#9699](matrix-org/synapse#9699))


Updates to the Docker image
---------------------------

- Speed up Docker builds and make it nicer to test against Complement while developing (install all dependencies before copying the project). ([\#9610](matrix-org/synapse#9610))
- Include [opencontainers labels](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) in the Docker image. ([\#9612](matrix-org/synapse#9612))


Improved Documentation
----------------------

- Clarify that `register_new_matrix_user` is present also when installed via non-pip package. ([\#9074](matrix-org/synapse#9074))
- Update source install documentation to mention platform prerequisites before the source install steps. ([\#9667](matrix-org/synapse#9667))
- Improve worker documentation for fallback/web auth endpoints. ([\#9679](matrix-org/synapse#9679))
- Update the sample configuration for OIDC authentication. ([\#9695](matrix-org/synapse#9695))


Internal Changes
----------------

- Preparatory steps for removing redundant `outlier` data from `event_json.internal_metadata` column. ([\#9411](matrix-org/synapse#9411))
- Add type hints to the caching module. ([\#9442](matrix-org/synapse#9442))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9499](matrix-org/synapse#9499), [\#9659](matrix-org/synapse#9659))
- Add additional type hints to the Homeserver object. ([\#9631](matrix-org/synapse#9631), [\#9638](matrix-org/synapse#9638), [\#9675](matrix-org/synapse#9675), [\#9681](matrix-org/synapse#9681))
- Only save remote cross-signing and device keys if they're different from the current ones. ([\#9634](matrix-org/synapse#9634))
- Rename storage function to fix spelling and not conflict with another function's name. ([\#9637](matrix-org/synapse#9637))
- Improve performance of federation catch up by sending the latest events in the room to the remote, rather than just the last event sent by the local server. ([\#9640](matrix-org/synapse#9640), [\#9664](matrix-org/synapse#9664))
- In the `federation_client` commandline client, stop automatically adding the URL prefix, so that servlets on other prefixes can be tested. ([\#9645](matrix-org/synapse#9645))
- In the `federation_client` commandline client, handle inline `signing_key`s in `homeserver.yaml`. ([\#9647](matrix-org/synapse#9647))
- Fixed some antipattern issues to improve code quality. ([\#9649](matrix-org/synapse#9649))
- Add a storage method for pulling all current user presence state from the database. ([\#9650](matrix-org/synapse#9650))
- Import `HomeServer` from the proper module. ([\#9665](matrix-org/synapse#9665))
- Increase default join ratelimiting burst rate. ([\#9674](matrix-org/synapse#9674))
- Add type hints to third party event rules and visibility modules. ([\#9676](matrix-org/synapse#9676))
- Bump mypy-zope to 0.2.13 to fix "Cannot determine consistent method resolution order (MRO)" errors when running mypy a second time. ([\#9678](matrix-org/synapse#9678))
- Use interpreter from `$PATH` via `/usr/bin/env` instead of absolute paths in various scripts. ([\#9689](matrix-org/synapse#9689))
- Make it possible to use `dmypy`. ([\#9692](matrix-org/synapse#9692))
- Suppress "CryptographyDeprecationWarning: int_from_bytes is deprecated". ([\#9698](matrix-org/synapse#9698))
- Use `dmypy run` in lint script for improved performance in type-checking while developing. ([\#9701](matrix-org/synapse#9701))
- Fix undetected mypy error when using Python 3.6. ([\#9703](matrix-org/synapse#9703))
- Fix type-checking CI on develop. ([\#9709](matrix-org/synapse#9709))


Synapse 1.30.1 (2021-03-26)
===========================

This release is identical to Synapse 1.30.0, with the exception of explicitly
setting a minimum version of Python's Cryptography library to ensure that users
of Synapse are protected from the recent [OpenSSL security advisories](https://mta.openssl.org/pipermail/openssl-announce/2021-March/000198.html),
especially CVE-2021-3449.

Note that Cryptography defaults to bundling its own statically linked copy of
OpenSSL, which means that you may not be protected by your operating system's
security updates.

It's also worth noting that Cryptography no longer supports Python 3.5, so
admins deploying to older environments may not be protected against this or
future vulnerabilities. Synapse will be dropping support for Python 3.5 at the
end of March.


Updates to the Docker image
---------------------------

- Ensure that the docker container has up to date versions of openssl. ([\#9697](matrix-org/synapse#9697))


Internal Changes
----------------

- Enforce that `cryptography` dependency is up to date to ensure it has the most recent openssl patches. ([\#9697](matrix-org/synapse#9697))


Synapse 1.30.0 (2021-03-22)
===========================

Note that this release deprecates the ability for appservices to
call `POST /_matrix/client/r0/register`  without the body parameter `type`. Appservice
developers should use a `type` value of `m.login.application_service` as
per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
type - will fail.


No significant changes.


Synapse 1.30.0rc1 (2021-03-16)
==============================

Features
--------

- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](matrix-org/synapse#9510), [\#9511](matrix-org/synapse#9511), [\#9573](matrix-org/synapse#9573))
- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](matrix-org/synapse#9540))
- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](matrix-org/synapse#9549))
- Optimise handling of incomplete room history for incoming federation. ([\#9601](matrix-org/synapse#9601))
- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9617](matrix-org/synapse#9617))
- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](matrix-org/synapse#9626))


Bugfixes
--------

- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](matrix-org/synapse#9473))
- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](matrix-org/synapse#9542), [\#9583](matrix-org/synapse#9583))
- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](matrix-org/synapse#9567))
- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](matrix-org/synapse#9587))
- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](matrix-org/synapse#9597))
- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](matrix-org/synapse#9620))
- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](matrix-org/synapse#9623))


Updates to the Docker image
---------------------------

- Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](matrix-org/synapse#8553))


Improved Documentation
----------------------

- Add relayd entry to reverse proxy example configurations. ([\#9508](matrix-org/synapse#9508))
- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](matrix-org/synapse#9550))
- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](matrix-org/synapse#9571))
- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](matrix-org/synapse#9580))
- Clarify the sample configuration for `stats` settings. ([\#9604](matrix-org/synapse#9604))


Deprecations and Removals
-------------------------

- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](matrix-org/synapse#9540))
- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](matrix-org/synapse#9559))


Internal Changes
----------------

- Add tests to ResponseCache. ([\#9458](matrix-org/synapse#9458))
- Add type hints to purge room and server notice admin API. ([\#9520](matrix-org/synapse#9520))
- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](matrix-org/synapse#9523))
- Fix incorrect type hints. ([\#9528](matrix-org/synapse#9528), [\#9543](matrix-org/synapse#9543), [\#9591](matrix-org/synapse#9591), [\#9608](matrix-org/synapse#9608), [\#9618](matrix-org/synapse#9618))
- Add an additional test for purging a room. ([\#9541](matrix-org/synapse#9541))
- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](matrix-org/synapse#9560))
- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](matrix-org/synapse#9561))
- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](matrix-org/synapse#9562))
- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](matrix-org/synapse#9563))
- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](matrix-org/synapse#9568))
- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](matrix-org/synapse#9576))
- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](matrix-org/synapse#9586))
- Add logging for redis connection setup. ([\#9590](matrix-org/synapse#9590))
- Improve logging when processing incoming transactions. ([\#9596](matrix-org/synapse#9596))
- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](matrix-org/synapse#9604))
- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](matrix-org/synapse#9619))
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Sep 1, 2021
Synapse 1.32.0 (2021-04-20)
===========================

**Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+.

This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities.

This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date.

Bugfixes
--------

- Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](matrix-org/synapse#9829))

Synapse 1.32.0rc1 (2021-04-13)
==============================

Features
--------

- Add a Synapse module for routing presence updates between users. ([\#9491](matrix-org/synapse#9491))
- Add an admin API to manage ratelimit for a specific user. ([\#9648](matrix-org/synapse#9648))
- Include request information in structured logging output. ([\#9654](matrix-org/synapse#9654))
- Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](matrix-org/synapse#9691))
- Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](matrix-org/synapse#9700))
- Add experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9717](matrix-org/synapse#9717), [\#9735](matrix-org/synapse#9735))
- Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](matrix-org/synapse#9710))
- Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](matrix-org/synapse#9766))

Bugfixes
--------

- Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](matrix-org/synapse#8926))
- Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](matrix-org/synapse#9711))
- Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](matrix-org/synapse#9725))
- Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](matrix-org/synapse#9770))
- Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](matrix-org/synapse#9780))

Updates to the Docker image
---------------------------

- Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](matrix-org/synapse#9765))

Improved Documentation
----------------------

- Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](matrix-org/synapse#9719))

Deprecations and Removals
-------------------------

- Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](matrix-org/synapse#9401))
- Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](matrix-org/synapse#9548))

Internal Changes
----------------

- Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](matrix-org/synapse#9718))
- Experiment with GitHub Actions for CI. ([\#9661](matrix-org/synapse#9661))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](matrix-org/synapse#9682))
- Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](matrix-org/synapse#9685))
- Improve Jaeger tracing for `to_device` messages. ([\#9686](matrix-org/synapse#9686))
- Add release helper script for automating part of the Synapse release process. ([\#9713](matrix-org/synapse#9713))
- Add type hints to expiring cache. ([\#9730](matrix-org/synapse#9730))
- Convert various testcases to `HomeserverTestCase`. ([\#9736](matrix-org/synapse#9736))
- Start linting mypy with `no_implicit_optional`. ([\#9742](matrix-org/synapse#9742))
- Add missing type hints to federation handler and server. ([\#9743](matrix-org/synapse#9743))
- Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](matrix-org/synapse#9753))
- Fix incompatibility with `tox` 2.5. ([\#9769](matrix-org/synapse#9769))
- Enable Complement tests for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary API. ([\#9771](matrix-org/synapse#9771))
- Use mock from the standard library instead of a separate package. ([\#9772](matrix-org/synapse#9772))
- Update Black configuration to target Python 3.6. ([\#9781](matrix-org/synapse#9781))
- Add option to skip unit tests when building Debian packages. ([\#9793](matrix-org/synapse#9793))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Z-Time-Tracked Element employees should track their time spent on this issue/PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants