This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Federation sender started spamming requests #16101
Assignees
Labels
A-Federation
A-Performance
Performance, both client-facing and admin-facing
O-Frequent
Affects or can be seen by most users regularly or impacts most users' first experience
S-Major
Major functionality / product severely impaired, no satisfactory workaround.
T-Defect
Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Comments
|
This just happened again, this time only one of the three federation senders and it started spamming two destinations. I had left
The Restarted a few times, the spam restarts when I type in #synapse:matrix.org or #element-web:matrix.org (the servers are participating in those rooms) A bit later when I didn't send anything in those rooms, one of the destinations updated, but the retry interval went way down for some reason edit: the next day, one of the servers still hasn't updated its retry_last_ts and is spamming requests again |
|
Looking at the ordinary logs a bit more (when it isn't spamming requests at 40hz), I noticed that my server is trying to send stuff to https://gist.github.com/tulir/99a8074ccf5e7dedbd7ae958ee2968d0 The destination says the last retry was ages ago and the retry interval is very big |
|
There's a suspicious increase (2-3x) in the background rate of failed transactions on July 3rd, which I think is when I updated to 1.87.0rc1
Beeper had an even more noticeable jump on August 1st (around when it updated from 1.85.2 to 1.88.0)
|
|
I have the same issue on 1.90.0rc1. When running Adding the configuration lines from the OP seems solve the issue for me at least. |
|
I can confirm that my redis is also getting spammed with those requests (one of the servers it started spamming requests to yesterday still hasn't recovered) |
|
looks like i have the same issue..
|
|
Manually updating |
can confirm it! |
|
More news: This seems to happen on 1.90.0rc1 without any workers or redis. There were other servers too obviously. Here's how many times this server has been hammered during the past ~18 hours (well, sort of): # grep pfaff.dev homeserver.log|wc -l
3077086 |
|
Just spotted #15743 in the 1.87.0 release notes which sounds suspicious, haven't found anything else that would seem related |
reivilibre
added
A-Federation
A-Performance
|
It looks like downgrading to 1.89.0 might "fix" the issue. So it could be #15743 combined with the changes between 1.89 and 1.90rc1. ....But also remember I was celebrating early when I thought adding the configuration lines from @tulir 's first post would fix the issue. :) I downgraded yesterday (Aug 14th). Monolithic Synapse (no workers or Redis). Also this is a single user server # grep federation -c homeserver.log*
homeserver.log:495866 (today)
homeserver.log.2023-08-12:603980
homeserver.log.2023-08-13:4555881
homeserver.log.2023-08-14:3794734Note: By "today" I mean the 17 hours the day's been going on since the logs rotate midnight. My larger Synapse HS log structure is more complicated so it was difficult to include. But downgrade seemed to help with that too. @reivilibre it's just three of us in this thread but could this be already considered a release blocker for 1.90.0? |
|
Same on my side after upgrade to 1.90.0... |
|
Two days into downgrading to 1.89.0. The spam hasn't started again; Neither with my monolithic Synapse nor my other server running with workers. |
|
On my side there is a spam from other servers to a domain on which I had some time ago (months) server for a few days, and I was the only user there with few chats joined. Now on the same wildcard domain, but different subdomain I have another server, and it is being spammed for a few days now with more and more requests each day on port 8448 ending in 401. After I made this old domain return 403 forcibly, it calmed down. I realize that dirty server/domain shutdown was not a good idea, but I did not expected such spam. |
|
This 401 response occurring because the server has been moved to a different server name (while the old server name still directly points to the homeserver) seems to be the common link across all these cases |
clokep
added
O-Frequent
|
Yeah, I am experiencing this now too. These are the top servers I see in my redis in about 1 minute: (Filtered with |
deepbluev7
added a commit
to deepbluev7/synapse
that referenced
this issue
Aug 22, 2023
This was broken in 1.90 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes matrix-org#16101 Signed-off-by: Nicolas Werner <[email protected]>
4 tasks
deepbluev7
added a commit
to deepbluev7/synapse
that referenced
this issue
Aug 22, 2023
This was broken in 1.90 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes matrix-org#16101 Signed-off-by: Nicolas Werner <[email protected]>
deepbluev7
added a commit
to deepbluev7/synapse
that referenced
this issue
Aug 22, 2023
This was broken in 1.87 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes matrix-org#16101 Signed-off-by: Nicolas Werner <[email protected]>
erikjohnston
pushed a commit
that referenced
this issue
Aug 23, 2023
…#16156) * Properly update retry_last_ts when hitting the maximum retry interval This was broken in 1.87 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes #16101 Signed-off-by: Nicolas Werner <[email protected]> * Add changelog * Change fix + add test * Add comment --------- Signed-off-by: Nicolas Werner <[email protected]> Co-authored-by: Mathieu Velten <[email protected]>
spantaleev
added a commit
to spantaleev/matrix-docker-ansible-deploy
that referenced
this issue
Aug 30, 2023
Hopefully fixes #2849 Related to: - matrix-org/synapse#16101 - matrix-org/synapse#16208
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
Two of my three federation senders are spamming two specific servers very quickly. Restarting them solved it temporarily, but they quickly started spamming the same two hosts again, possibly after sending an event to a room the hosts were in. After that I tried adding
to the config and restarting. After that it didn't start spamming again, but I'm not sure if it was the config or if it just wanted to be restarted twice, as it had updated the
destinationstable with a newretry_last_tsandretry_interval(the new interval being the maximum)Homeserver
maunium.net
Synapse Version
1.90.0rc1
Database
PostgreSQL
Workers
Multiple workers
Platform
Docker (custom image)
Relevant log output
The text was updated successfully, but these errors were encountered: