After a user redacted their own join event from HQ, Android DoSes us with /context requests

[ara4n]

We saw 2061 requests to '/_matrix/client/r0/rooms/%21cURbafjkfsMDVwdRDQ%3Amatrix.org/context/%2415153218301304484Fpgds%3Amatrix.org?limit=1' from Riot/Android clients after a user redacted that event (they sent 3 redactions, as presumably it didn't seem to work). This looks to have contributed to taking out the main synapse process for ~30 minutes

[ara4n]

^ the carnage this caused on CPU on the context servlet

[ylecollen]

The main problem is the performance of room state events redaction
-> either the client lets the server does the job : it looks weird that only 2061 requests are enough to stuck the server for 30 minutes
-> or the android stores the state events. It means to save about 13000 state events for Matrix HQ, load them in redaction case and recompute the new room state status.
On low / middle range devices, the storage use explodes + many devices already reported that the application crashed because of the 13000 events loading in RAM.

[ara4n]

why is it calling /context at all, though? the right API to recompute the state is /state (which may be less heavy)?

[ylecollen]

@ara4n

For the both mobile clients,

When an event is redacted (from the sync)

-> check if the event is known, if it is a room state, compute the new room state from the state events list stored in the application store (disabled on Android because of out of memory crashes)

-> if the event is not known, a "getContent of event" request is triggered to retrieve the message type (this is your server overload issue). If a room state event, we perform a room initial sync (because we don't know which updates it implies).

The context request is done because there is no other way to retrieve a message content from its room id / event id.

[richvdh]

(the agreed solution is to implement /rooms/{roomId}/event/{eventId} on the server and have the mobile client use that instead of the more heavy-weight .../context/...)

[ylecollen]

@richvdh @ara4n
getContextOfEvent is replaced by getEvent which
1- tries /rooms/{roomId}/event/{eventId}
2- if the new request is not supported, tries /events/{eventId} (which still works)
3- if "/events/{eventId} " is not supported, use the /context request

we need to be backward compatible until all the servers are updated.