Replication gets stuck with "stream X has fallen behind"

[richvdh]

If a worker gets sufficiently behind that there are more than 10000 updates for it to catch up on, the master drops the replication connection, and there is no way for it to catch up again.

The master reports Exception: stream <stream> has fallen behind.

Related: #4388.

[richvdh]

The workers do actually make progress through the backlog - it's just slower than the arrival of new stuff (they get about 400 events on each connection, and make a new connection about every 30s).

One solution might to make the replication protocol distinguish between "catching up" (where the replication client requests batches of updates starting at a certain stream ID, and it will not get disconnected if it gets "behind") and "streaming" (where the server attempts to send updates in realtime).

[richvdh]

I think this relates only to old-style TCP replication, which per #11728 we should get rid of anyway, so closing as a WONTFIX