0.99.5.1 stopped working

[kroeckx]

I was having problems that it's using lots of RAM (12 GB) and CPU, so someone pointed me to #1760 (comment). I had some counts that were higher than 100, so I ran that query. Afterwards it stopped using that much RAM, it's only using 1.2 GB now. Synapse is still using 100% CPU, and postgres is only using about 1% CPU now instead of 150%.

But at least federation only worked for like the first 10 minutes. I'm now not getting any messages back, I'm not sure my messages make it. I'm the only user, so it's hard to see. I guess I could test on one of the channels that are bridged to IRC.

[erikjohnston]

Probably caused by #5269

[kroeckx]

I have applied the patch in #5274, which has been running for 20 minutes now, but I still don't get any new messages.

[erikjohnston]

That's probably because the extremities have already stacked up, as #5274 only fixes things going forwards. I'm currently writing a background update that should hopefully clear those.

[kroeckx]

Note I ran that query that I linked to, which removed all but 1 of the extremities as I understand it.

[erikjohnston]

Oh, hmm, sorry misread that. That does indeed sound like something else.

[kroeckx]

So it seems that the messages I send do make it to other servers, I just don't get anything back.

[kroeckx]

Any suggestion on what I can do to fix things, or how to provide more information so that it can be fixed?

[erikjohnston]

I don't suppose you have a prometheus hooked up to it? (I was hoping that us fixing some of the issues that matrix.org deployment is having recently would find whatever issue other people are running into)

[kroeckx]

I don't, but if you tell me what kind of data you want I can provide it.

[erikjohnston]

Broadly, often the easiest way of tracking this stuff down is to just stare at the dashboard a bit and see if anything stands out. Otherwise, running pyflame (if you're not running py3.7) can often give a good snapshot of what it's doing. py-spy works on py3.7 (though use --nonblocking option)

[kroeckx]

I've put the output of py-spy on:
https://www.roeckx.be/synapse.svg

[kroeckx]

It seems I just needed to restart riot and that fixed everything.