Obscure exceptions when TLS cert on SMTP server is untrusted

[richvdh]

I think this is happening because my SMTP server is presenting a self-signed cert (and I can't stop synapse trying to STARTTLS, per #8046)

2021-03-09 10:16:38,776 - synapse.push.emailpusher - 147 - ERROR - emailpush.process-15454 - Exception processing notifs
Capture point (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/homeserver.py", line 507, in <module>
    main()
  File "/home/synapse/src/synapse/app/homeserver.py", line 503, in main
    run(hs)
  File "/home/synapse/src/synapse/app/homeserver.py", line 494, in run
    logger=logger,
  File "/home/synapse/src/synapse/app/_base.py", line 135, in start_reactor
    run()
  File "/home/synapse/src/synapse/app/_base.py", line 119, in run
    run_command()
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/base.py", line 1423, in run
    self.mainLoop()
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/base.py", line 1433, in mainLoop
    reactorBaseSelf.runUntilCurrent()
  File "/home/synapse/src/synapse/metrics/__init__.py", line 548, in f
    ret = func(*args, **kwargs)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/base.py", line 999, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/tcp.py", line 507, in connectionLost
    self.connector.connectionLost(reason)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/base.py", line 1312, in connectionLost
    self.factory.clientConnectionLost(self, reason)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/mail/smtp.py", line 1935, in clientConnectionLost
    self._processConnectionError(connector, err)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/mail/smtp.py", line 1952, in _processConnectionError
    self.result.errback(err.value)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 517, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 580, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1511, in gotResult
    current_context.run(_inlineCallbacks, r, g, status)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks
    result = current_context.run(result.throwExceptionIntoGenerator, g)
  File "/home/synapse/env/lib/python3.7/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
Traceback (most recent call last):
  File "/home/synapse/src/synapse/push/emailpusher.py", line 145, in _process
    await self._unsafe_process()
  File "/home/synapse/src/synapse/push/emailpusher.py", line 195, in _unsafe_process
    await self.send_notification(unprocessed, reason)
  File "/home/synapse/src/synapse/push/emailpusher.py", line 312, in send_notification
    self.app_id, self.user_id, self.email, push_actions, reason
  File "/home/synapse/src/synapse/push/mailer.py", line 308, in send_notification_mail
    await self.send_email(email_address, summary_text, template_vars)
  File "/home/synapse/src/synapse/push/mailer.py", line 360, in send_email
    requireTransportSecurity=self.hs.config.require_transport_security,
twisted.internet.error.ConnectionAborted: Connection was aborted locally using ITCPTransport.abortConnection.

[anoadragon453]

I suppose try fixing the cert and see if the issue persists?

Edit: Ah, this is for local testing. That is a bit trickier.

[richvdh]

my complaint is not the fact that there's an error, so much as the fact that the error is unnecessarily verbose and yet tells me nothing about what the problem is.

[anoadragon453]

We could except on that Exception. I'm not sure whether it would only be TLS issues that would raise it though. But we could put a generic failure message in that would at least remove the large, useless trackback.

[richvdh]

right. Isn't there a way to tell if TLS was the problem? If not, that might be a thing we should fix in Twisted.

[hmenke]

I have a very similar, if not the same problem:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/identity.py", line 382, in send_threepid_validation
    await send_email_func(email_address, token, client_secret, session_id)
  File "/usr/local/lib/python3.8/site-packages/synapse/push/mailer.py", line 207, in send_add_threepid_mail
    await self.send_email(
  File "/usr/local/lib/python3.8/site-packages/synapse/push/mailer.py", line 349, in send_email
    await make_deferred_yieldable(
twisted.internet.error.ConnectionAborted: Connection was aborted locally using ITCPTransport.abortConnection.

I would suggest adding an option

mail:
  skip_verify: true

to accept self-signed certificates.

---

As a workaround I tried adding the certificate of my mailserver to the Docker image using

FROM matrixdotorg/synapse:v1.29.0

RUN mkdir -p /usr/local/share/ca-certificates \
  && echo quit | \
    openssl s_client -connect mail.example.com:587 -starttls smtp | \
    openssl x509 -outform PEM -out /usr/local/share/ca-certificates/mail.example.com.crt \
  && update-ca-certificates

which makes OpenSSL accept the certificate, but Twisted doesn't seem to use OpenSSL's trust store.

[richvdh]

I'd really like to keep this issue focussing on the incomprehensible exception, rather than feature-creeping it into disabling TLS or whatever.

[thegcat]

I have a data point to add to this issue, and depending on whether this fixes the original issue or not might be a separate one… Anyway.

We debugged a problem leading to similar issues as presented above in our setup, and have found out that twisted insists on the certificate presented by the mail-server on STARTTLS being issued to the hostname twisted/synapse uses to connect to said mail server. This change stems from twisted/twisted#1225, the new context comes from https://twistedmatrix.com/documents/18.4.0/api/twisted.internet.ssl.optionsForClientTLS.html which stipulates:

first, and most importantly, it verifies that the certificate received from the server correctly identifies the specified hostname

This might also be interesting for the configuration documentation, as the default localhost will probably not work in most cases where the mail server also does STARTSSL.

[RenWal]

Can confirm that this breaks the default configuration. The only way to stop this from happening right now seems to either be to put in the effort of generating a valid TLS certificate for localhost using your own CA, or changing the mail server config to disallow STARTTLS on the loopback interface completely. For postfix, you can do it by editing /etc/postfix/master.cf like so:

<your external interface>:smtp      inet  n       -       y       -       -       smtpd
127.0.0.1:smtp inet n       -       y       -       -       smtpd
  -o smtpd_tls_security_level=none

[richvdh]

I've raised https://twistedmatrix.com/trac/ticket/10210 to track this on the Twisted side.

[richvdh]

for the record: as of #10546, you can disable TLS for the SMTP connection.