Twisted: Bad Descriptor/Malformed Descriptor Found

[Casuallynoted]

Description

The server has consistent downtime, usually lasting about a minute. Then it begins working again as it should. This appears to be due to an issue with Twisted. I'm not entirely sure if this is a bug or a support issue, but I have asked in the support chat and haven't been able to find anything out and have also triple checked my setup so I believe it may be a bug, potentially with Synapse on MacOS/M1 ARM chips as that is what I am running the server on and have tested it with.

Steps to reproduce

• Start server, allow to run for some time
• Keep an eye on logs, issue begins to happen fairly quickly
• During the downtime, all connections to the server are not made correctly- including those to and from the reverse proxy

I've included the logs I'm looking at with this issue to see if this can help narrow things down. If this is not a bug, I do apologise but I think it could be: https://pastebin.com/W0uKH37G

Version information

• Homeserver: Another homeserver, chat.casually.cat

If not matrix.org:

• Version: 1.24.0 with Python 3.9.1

• Install method:

Installed via pip, exclusively followed the git install guide here: https://github.com/matrix-org/synapse/blob/develop/INSTALL.md

• Platform: Apple Silicon (M1) Mac running Mac OS 11.1 Big Sur

The homeserver is running natively (not in a vm/container) by entering a python env and running synctl start in the app directory.

[clokep]

The only reference to this error I can find on the Twisted issue tracker is https://twistedmatrix.com/trac/ticket/8174 (not directly related). I wonder if you're running out of file descriptors?

See https://github.com/matrix-org/synapse#running-out-of-file-handles

[Casuallynoted]

The only reference to this error I can find on the Twisted issue tracker is https://twistedmatrix.com/trac/ticket/8174 (not directly related). I wonder if you're running out of file descriptors?

See https://github.com/matrix-org/synapse#running-out-of-file-handles

I did look into that one but my file handles are set to 10240 currently, which is well above the recommended amount and also the maximum amount that my system can handle. It's still possible that that could be it but if so I wouldn't be sure how to correct it.

[clokep]

I'd see if anyone else in#synapse:matrix.org is running on an M1. If not maybe file a bug against Twisted?

[Casuallynoted]

I'd see if anyone else in#synapse:matrix.org is running on an M1. If not maybe file a bug against Twisted?

Looks like no M1 folks in there, but I did create an issue with Twisted so I'm hoping maybe that gets some traction. It seems like CORS requests are frequently being denied as well. I've noticed this really kills the synchronization between any devices running synapse clients, and also kills my ability to use encrypted rooms because oftentimes folks can't consistently receive the encryption keys for my messages.

This is the ticket at Twisted's website as well just for reference-
https://twistedmatrix.com/trac/ticket/10076#ticket

[clokep]

Thanks for filing an issue with Twisted! Without additional information my hunch is that this is an upstream issue. I'm not sure what to suggest trying, maybe #8642 would be a difficult workaround?

[Casuallynoted]

Thanks for filing an issue with Twisted! Without additional information my hunch is that this is an upstream issue. I'm not sure what to suggest trying, maybe #8642 would be a difficult workaround?

I'm absolutely down to give it a shot! Would I just need to run synapse with the environment variable mentioned in the last comment? Or are there other steps to run it that way?

[clokep]

Thanks for filing an issue with Twisted! Without additional information my hunch is that this is an upstream issue. I'm not sure what to suggest trying, maybe #8642 would be a difficult workaround?

I'm absolutely down to give it a shot! Would I just need to run synapse with the environment variable mentioned in the last comment? Or are there other steps to run it that way?

It would require running an unreleased Twisted & modifying Synapse unfortunately, so quite a bit complicated.

[Casuallynoted]

Thanks for filing an issue with Twisted! Without additional information my hunch is that this is an upstream issue. I'm not sure what to suggest trying, maybe #8642 would be a difficult workaround?

I'm absolutely down to give it a shot! Would I just need to run synapse with the environment variable mentioned in the last comment? Or are there other steps to run it that way?

It would require running an unreleased Twisted & modifying Synapse unfortunately, so quite a bit complicated.

Oof, yikes ^^; While I'm tech savvy enough to set up the server as-is, I'm not sure I'd be able to do that. I wish I could, it would be a great way to troubleshoot for sure. I majorly appreciate the suggestion too!

I really hope I can get it up and running, I've been wanting to switch to Matrix full-time for over a year but with the way mine is right now it just doesn't work well enough to be used on the daily. Hopefully Twisted will get back eventually with some info

[ShadowJonathan]

Thanks for filing an issue with Twisted! Without additional information my hunch is that this is an upstream issue. I'm not sure what to suggest trying, maybe #8642 would be a difficult workaround?

I'm absolutely down to give it a shot! Would I just need to run synapse with the environment variable mentioned in the last comment? Or are there other steps to run it that way?

Right now it is not possible, that issue is blocked due to twisted not releasing a version in which twisted/twisted#1237 is included, since last year's march.

I'm currently waiting patiently for twisted to drop another release, so I could either potentially up the requirements, or create a package .[extra] so i can shove the dependency requirement in there.