Media sync of room joined over federation causes infinite loop of download if remote file size > local file size limit

[skepticalwaves]

Description

We recently increased our complexity limit for federated rooms and someone joined a matrix.org room. The server began to aggressively cache the media store, and arrived at file with size larger than the 10M limit we had set. This resulted in the following log entries over and over again, a massive cpu and network load, and general unpleasantness.

I resolved it by setting the limit to 50M and restarted synapse.

Apr 01 21:15:22 matrix matrix-synapse[2006692]: 2021-04-01 21:15:22,832 - synapse.rest.media.v1.media_repository - 419 - WARNING - GET-165511 - Failed to fetch remote media matrix.org/WxMHVWcDufQmDWStcjuWODHf
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,156 - synapse.http.matrixfederationclient - 1002 - WARNING - GET-165554 - {GET-O-12318} [matrix.org] Requested file is too large > 10485760 bytes
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,157 - synapse.rest.media.v1.media_repository - 419 - WARNING - GET-165554 - Failed to fetch remote media matrix.org/WxMHVWcDufQmDWStcjuWODHf
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,479 - synapse.http.matrixfederationclient - 1002 - WARNING - GET-165565 - {GET-O-12319} [matrix.org] Requested file is too large > 10485760 bytes
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,480 - synapse.rest.media.v1.media_repository - 419 - WARNING - GET-165565 - Failed to fetch remote media matrix.org/WxMHVWcDufQmDWStcjuWODHf
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,994 - synapse.http.matrixfederationclient - 1002 - WARNING - GET-165580 - {GET-O-12320} [matrix.org] Requested file is too large > 10485760 bytes
Apr 01 21:15:23 matrix matrix-synapse[2006692]: 2021-04-01 21:15:23,995 - synapse.rest.media.v1.media_repository - 419 - WARNING - GET-165580 - Failed to fetch remote media matrix.org/WxMHVWcDufQmDWStcjuWODHf
Apr 01 21:15:24 matrix matrix-synapse[2006692]: 2021-04-01 21:15:24,363 - synapse.http.matrixfederationclient - 1002 - WARNING - GET-165599 - {GET-O-12321} [matrix.org] Requested file is too large > 10485760 bytes
Apr 01 21:15:24 matrix matrix-synapse[2006692]: 2021-04-01 21:15:24,364 - synapse.rest.media.v1.media_repository - 419 - WARNING - GET-165599 - Failed to fetch remote media matrix.org/WxMHVWcDufQmDWStcjuWODHf
Apr 01 21:15:25 matrix matrix-synapse[2006692]: 2021-04-01 21:15:25,108 - synapse.http.matrixfederationclient - 1002 - WARNING - GET-165602 - {GET-O-12322} [matrix.org] Requested file is too large > 10485760 bytes

Steps to reproduce

• Have a low limit for file size
• Join a federated room with files large than this size

Version information

synapse in docker matrixdotorg/synapse:v1.30.0 deployed using https://github.com/spantaleev/matrix-docker-ansible-deploy

Platform:
Ubuntu 20.04 with docker-ce 5:20.10.5~3-0~ubuntu-focal

[clokep]

I think this is pretty much a duplicate of #9453 -- Synapse doesn't remember that it previously tried to download this and found it to be too large.

[skepticalwaves]

Indeed this looks very similar. I can't tell if its exactly the same codepaths.

[richvdh]

closing as a dup of #9453