Calling the deactivation API without rate limiting causes room inconsistencies

[ThoreKr]

Description

When bulk removing users in the IdM (keycloak), at the same time multiple requests will be sent to the /_synapse/admin/v1/deactivate/<user_id> endpoint to deactivate (erase=true) the concerning accounts in synapse.

When not rate limiting these delete requests, the erasure can have the following side effect for a subset of the deactivated users:

• user unset their display name
• user left the room
• user joined the room

These hulls of deactivated accounts then sit in the room as regular members. Manually calling the deactivate endpoint again will remove them from the room again, but they shouldn't be rejoining in the first place.

Steps to reproduce

• Have a room with about 1k users (authenticated via OIDC) all auto-joining into a shared public, non-encrypted room
• In a for loop without wait times call the deactivate endpoint for 20-30 users
• Watch them unsetting their display name and leaving
• Some will shortly after rejoin the room

Homeserver

selfhosted

Synapse Version

1.89.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgresql, single instance

Workers

Single process

Platform

x86 (64 bit) VM

Configuration

• Presence is enabled
• Message retention is enabled

  purge_jobs:
    - longest_max_lifetime: 2d
      interval: 1h
    - shortest_max_lifetime: 2d
      interval: 1d

Relevant log output

Only the api call to the deactivate endpoint is logged.

Anything else that would be useful to know?

No response

[reivilibre]

possibly related: if we deduplicated deactivations, it might sort out the issues seen here — #16055

[erikjohnston]

I think we can also probably just skip setting peoples display names in rooms when doing the deactivation? That would fix the race and reduce the cost of deactivation.

I think that would be a case of only calling the following on deactivation?

synapse/synapse/handlers/profile.py

Line 190 in 2763c49

await self._update_join_states(requester, target_user)

[clokep]

I think we can also probably just skip setting peoples display names in rooms when doing the deactivation?

Or could we do it as a background task?