replication wedged after spike in event traffic

[richvdh]

we received a big spike of 10000 membership events; afterwards replication seemed to get itself into a loop of repeated 'could not keep up' disconnections.

[turt2live]

Related: #3495 ?
Related: #2738

[richvdh]

I don't think it was due to send_join requests, so #3495 is likely different. It could be related to #2738, though in our case the synapse master was pinned to 100% CPU so again it looked a bit different.

[richvdh]

Master CPU graph. Influx of events was at 07:13. Note that it handles that ok and doesn't actually blow up until 07:20. The dip at 07:34 was a whole-system restart.

[richvdh]

It's all going to replication-REPLICATE-caches

I guess the problem is this big spike in cache invalidation commands

It doesn't actually seem to be making much progess, though

[richvdh]

I'm going to assume that this is no longer useful, given replication has been rewritten.