Events sent immediately after joining can be incorrectly soft-failed

[matrixbot]

This issue has been migrated from #10066.

---

Description

The Gitter bridge seems to have reproduced a race condition in federation where because the appservice joins and sends a message in quick succession, the message can out run the join which causes the message to soft_fail.

To see an actual reproduction, there is a missing message in the !SrkiFczPSWmYrlSNYF:matrix.org room for matrix.org homeserver users ($mMJdjGPC2GaoWK3pLApa_O2vBoeuQwYtP6iY6XzbrfY). This is tracked on Gitter by https://gitlab.com/gitterHQ/webapp/-/issues/2770#note_583925093

The event exists in the gitter.im homeserver database and is visible to the gitter.im homeserver where the Gitter bridge appservice operates. The message is also visible to any new homeserver that comes in the room and backfills the messages. It's only not visible on the matrix.org homeserver.

The message also exists on the matrix.org homeserver database but is soft_failed (thanks to @richvdh for checking the database)

@Half-Shot has also seen this happen a few times with the IRC bridge.

As @leonerd from matrix-org/synapse#1444 and @Half-Shot mentioned, this is probably not a problem for normal users because the time taken to join and then send a message is some number of seconds. Whereas with appservices, can join and send an event in quick succession (almost instantly).

Related issues:

• events sent immediately after joining can be incorrectly soft-failed matrix-org/synapse#6536: events sent immediately after joining can be incorrectly soft-failed
• events get soft-failed when the federation_inbound worker is busy matrix-org/synapse#7744: events get soft-failed when the federation_inbound worker is busy
• Setting profile + joining a room can race (SYN-787) matrix-org/synapse#1444: Reproduces with a join, set profile, membership event (not over federation though).
• Soft-failures make federated bans racy and frustrate redaction matrix-org/synapse#9329

Steps to reproduce

1. Create a room on HS1, !my-room:hs1
2. Setup HS2 with an appservice
3. Join and send a message through the appservice to !my-room:hs1
   • The appservice is probably not necessary to reproduce. Just send a join and message event from HS2
4. Check whether the message is visible on hs1
5. Since it's a race condition, this probably does not always reproduce

Version information

• Homeserver: gitter.im -> matrix.org