Synapse pegs cpu at 100% and disconnects all clients

[davehayes]

Description

Two days in a row I have gone to read the room history for rooms I am interested in and suddenly synapse will disconnect all clients and spin 100%, filling up log files with lots of stack trace and error messages. I have attempted to extract the meaningful bits here but of course there's a big wall of log text that I may have missed:

# ls -lat
total 1082888
-rw-r-----  1 synapse  synapse   59915573 Sep 26 12:36 homeserver.log
drwxr-xr-x  2 synapse  synapse        512 Sep 26 12:19 .
-rw-r-----  1 synapse  synapse  104857584 Sep 26 12:18 homeserver.log.1
-rw-r-----  1 synapse  synapse  104857479 Sep 26 12:16 homeserver.log.2
drwxr-xr-x  8 root     wheel         3072 Sep 26 11:00 ..
-rw-r-----  1 synapse  synapse  104857252 Sep 26 07:34 homeserver.log.3

I should also mention I have encoding: utf8 in my log.config.

Steps to reproduce

• I am unable to reproduce this

Version information

• Homeserver:

matrix.jetcafe.org

• Version:

# curl https://matrix.jetcafe.org:8448/_synapse/admin/v1/server_version
{
    "python_version": "3.6.9",
    "server_version": "1.2.1"
}

• Install method:

FreeBSD ports

• Platform:

FreeBSD 11.3 r349638, dedicated bare metal. I'm the package builder for this server so feel free to ask any questions.

[richvdh]

ugh. That sounds a lot like #4240, which we never really got to the bottom of.

[davehayes]

Indeed. I read that with interest. I'm wondering if there is a way I can help debug it, but this may be problematic because I build synapse through FreeBSD ports. Still, I'm the one who builds it directly (I don't use FreeBSD's official package repos) so...I'm willing to help at any rate! :D

[seth586]

Same problem here, compiled from ports, only change was I added urlpreview to the makefile.

Happens about once a day, for now I'll just write a script to restart if CPU hits 100 ...

[clokep]

The first thing to check is that Synapse is up-to-date and that the logging config is up-to-date with the Synapse repo since there were some fixes in this area.

[seth586]

Good catch, although I am running v1.26.0

handlers:
    file:
        class: logging.handlers.TimedRotatingFileHandler
        formatter: precise
        filename: /var/log/matrix-synapse/homeserver.log
        when: midnight

The lockup happened roughly around midnight system UTC time, and I was writing logs to /dev/null.

To test this, what values can I substitute for midnight to represent a certain time?

Is there a more eloquent way to disable logging?

[clokep]

Is there a more eloquent way to disable logging?

I can't really recommend doing that, but removing all handlers will probably do that.

[richvdh]

Given the lack of follow-up here, I'm going to assume the issue is fixed.