retry device resync doesn't follow exponentially back off algorithm

[matrixbot]

This issue has been migrated from #9603.

---

Description

As far as I'm aware matrix-synapse should stop syncing to dead servers using an exponential back off algorithm.
Reference:
matrix-org/synapse#4979
matrix-org/synapse#5113

This actually doesn't happen if the error is "No Content-Type header received from remote server". Most likely the server owner disabled the server and left the reverse proxy online or the domain got resold and now there is another HTTP/HTTPS service on that domain.

In my case my server tries to contact it almost twice a minute and is spamming my logs. The exponential back off algorithm does not work in that case.

Logs in my case:

`2021-03-12 12:12:11,582 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1834 - {GET-O-15116} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:11,583 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1834 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,812 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1835 - {GET-O-15120} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,813 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1835 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1836 - {GET-O-15126} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1836 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1837 - {GET-O-15130} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1837 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,813 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1838 - {GET-O-15134} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1838 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1839 - {GET-O-15138} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1839 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1840 - {GET-O-15142} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1840 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server

Steps to reproduce

• install synapse
• find a user whose server is returning no content-type header
• check logs

Version information

• Homeserver:

• Version: 1.29.0+buster1

• Install method: Debian Package Manager

• Platform: Debian, LXC Container

[xundeenergie]

My logfiles are filled with messages like

2024-09-10 08:07:42,972 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @orloskynedi:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:42,994 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5059} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40feliciaabebi%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:42,994 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @feliciaabebi:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,014 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5060} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40glanville_luisa%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,014 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @glanville_luisa:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,037 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5061} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40thadeusnepean%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:43,037 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @thadeusnepean:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,060 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5062} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40grider_kellsie%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,061 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @grider_kellsie:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,082 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5063} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40kaceygabriello%3Anyctico.de: HttpResponseException('301: Moved Permanently')

It starts every 20 to 30 seconds.

Version: matrix-synapse-py3 1.114.0+bookworm1

[AndrewKvalheim]

As an admin is there anything I can do manually to make Synapse forget these and stop trying?