Trouble with using local_certificate_path & local_key_path

[gerneio]

Unfortunately the email client I am using is hardcoded to enable SSL (.NET SmtpClient embedded within 3rd parties software), so I must have a secure connection between the client and the proxy. However, I am having a difficult time getting this to work with the proxy and could use some guidance.

I used mkcert to create a cert for local dev & testing (mkcert -key-file key.pem -cert-file cert.pem localhost 127.0.0.1). I then altered emailproxy.config accordingly:

[SMTP-1587]
server_address = smtp.office365.com
server_port = 587
starttls = True
local_certificate_path = ./config/cert.pem
local_key_path = ./config/key.pem

And when attempting to connect the mail client to the email proxy, it hangs for a bit until the MAX_SSL_HANDSHAKE_ATTEMPTS limit is reached. Here are the logs (note that I added some additional log points to get the SSL specific errors in case they were useful):

SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) <-> [ Starting TLS handshake ]
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) --> [ Client connected ]
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587)     <-- b'220 DS7P222CA0017.outlook.office365.com Microsoft ESMTP MAIL Service ready at Thu, 4 Jan 2024 17:48:46 +0000\r\n'
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) SMTPModifySenderPlugin : receive_from_server --> b'220 DS7P222CA0017.outlook.office365.com Microsoft ESMTP MAIL Service ready at Thu, 4 Jan 2024 17:48:46 +0000\r\n'
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) <-- b'220 DS7P222CA0017.outlook.office365.com Microsoft ESMTP MAIL Service ready at Thu, 4 Jan 2024 17:48:46 +0000\r\n'
... (below two lines repeat multiple times until `MAX_SSL_HANDSHAKE_ATTEMPTS` is reached)
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) [ TLS handshake started... ]
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) [ SSLWantReadError ] The operation did not complete (read) (_ssl.c:1000)
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) [ TLS handshake started... ]
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) [ SSLWantReadError ] The operation did not complete (read) (_ssl.c:1000)
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) SSL socket handshake failed (reached `MAX_SSL_HANDSHAKE_ATTEMPTS`)
Caught connection error in SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) : <class 'ssl.SSLError'> with message: ac.robinson.email-oauth2-proxy
Is the server's `starttls` setting correct? Current value: True
You have set `local_certificate_path` and `local_key_path`: is your client using a secure connection? github.com/FiloSottile/mkcert is highly recommended for local self-signed certificates, but these may still need an exception in your client
If you encounter this error repeatedly, please check that you have correctly configured python root certificates; see: https://github.com/simonrob/email-oauth2-proxy/issues/14
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) --> [ Client disconnected ]
SMTP ([::1]:52606-{[::1]:1587}-smtp.office365.com:587) <-- [ Server disconnected ]

So it would seem the TLS handshake is failing with SSLWantReadError: The operation did not complete (read), but have not been able to figure out why exactly.

Note that I tried a few different email clients with SSL enabled (.NET's SmtpClient, powershell's Send-MailMessage, python's smtplib.SMTP, as well as openssl cmd line utility), all of which produced similar logs as shown above.

FYI, I used the following openssl cmd for testing: openssl s_client -starttls smtp -ign_eof -crlf -connect localhost:1587, which seemed to work fine if I pointed it directly to smtp.office365.com:587 (for example). This article was a good guide for how to test sending SMTP commands manually.

I'm kind of at a loss for why it's not working with these local dev certs. As a quick test, I did setup a simple python script that opened a secure socket with the same cert. and it did seem to work. But I know there's a little more involved when it comes to an SMTP connection w/ TLS and such.

Appreciate any direction you can provide.

[gerneio]

Forgot to mention that I did review issue #14 and followed steps for updating the python root certificates, however it didn't seem to make any difference (I'm testing on a windows 11 laptop)

[simonrob]

Is your client configured to trust self-signed certificates? Many applications/clients are fine, when using mkcert but others (for example, Firefox) require an exception to be set up.

What is the output when you enable the proxy's debug mode and try the following command in a terminal? (Replacing with your own IP and port)

openssl s_client -crlf -connect 127.0.0.1:1588

[simonrob]

Were you able to make any progress with this?

[gerneio]

I have not been able to test any further. The openssl command you suggested running (i.e. w/o starttls/smtp) I'm fairly certain I've already tested, and IIRC it just hangs since the starttls command is expected by the proxy, but never received. Would need to retest to confirm that behavior, but fairly certain that's accurate.

As for making sure the client is configured to trust self-signed certificates, the only "clients" I'm using are python, powershell, and .net scripts, as well as the openssl command, which, to my knowledge, dont need to be configured to accept selft-signed certs (correct me if I'm wrong). Since I couldn't get those other scripts to work, I focused on trying to get it to work with openssl directly, since if I can't get it to work at that level, then of course no script or client is going to work. As mentioned, I can get the openssl command to work against Office 365's smtp endpoint, just not with the proxy. I haven't moved to testing in any actual email client program, as I'd expect to get it to work with openssl command first, therefore negating the allowing the self signed certs (I believe).

Are you actively able to get this to work with mkcert and openssl command?

[simonrob]

Please could you elaborate about your mention of STARTTLS? This could well be the issue – you should not enable STARTTLS in your client, because as explained in the example configuration file the proxy must handle this itself in order to be able to intercept the authentication commands on your behalf.

Re: certificates, just for completeness here is the full list of commands that you'd run to set this up from scratch:

brew install mkcert
mkcert -install
mkcert 127.0.0.1

Use the following configuration file entry with the proxy:

[SMTP-1587]
server_address = smtp.office365.com
server_port = 587
starttls = True
local_certificate_path = /path/to/127.0.0.1.pem
local_key_path = /path/to/127.0.0.1-key.pem

Then run the proxy (python emailproxy.py) and SMTP works without issue using the following OpenSSL command:

openssl s_client -crlf -connect 127.0.0.1:1587

[gerneio]

Quick follow-up:

I started from scratch and followed your steps exactly from above, but this time I stayed within a linux environment (been using windows, but not sure that's going to make a big difference in the long run).

Only step I had to modify was with the last part when using the OpenSSL command: openssl s_client -ign_eof -crlf -connect 127.0.0.1:1587. The ign_eof just helped to allow subsequent text commands from the command window with line-endings to be sent and received properly (was failing when I get to the RCPT TO part). Afterwards, I was finally able to get an email sent successfully using the OpenSSL command.

I tried my other scripting email clients (.NET's SmtpClient & powershell's Send-MailMessage) but they were still failing with the SSL socket handshake failed (reached MAX_SSL_HANDSHAKE_ATTEMPTS) error. I tried a different .NET library called MailKit, and was able to get it to work though. The third party program I will ultimately end up needing to use this with is using the .NET SmtpClient, which is hardcoded with EnableSsl=True. I have no ability to change the client that is used or whether or not the EnableSsl property is on or off (otherwise I wouldn't of brought up this issue).

My next steps are to decipher that when setting EnableSsl=True, how does that affect what commands the .NET client sends over to the proxy. I suspect that when enabled, the .NET client will start to perform the STARTTLS negotiations, which I can't disable. I will confirm when I know more.

Not sure what platform you're running on, but .NET is cross-platform now, so if you're feeling up for helping get to the bottom of it, you can test this same code anywhere (I believe). For testing, I'd recommend installing powershell core, since it will include .NET libs. Attached are three powershell scripts I've been using for testing this out. The Mailkit script is the one that does seem to work out-of-the-box, and the other two do not. IIRC, the Send-MailMessage one basically just sends as the .NET SmtpClient underneath.

SendTestEmail_Scripts.zip

[simonrob]

hardcoded with EnableSsl=True

This is the issue. You mentioned previously that SSL was hardcoded, but that link provides the extra context: for some reason this client only supports the STARTTLS method, and does not support connections that are SSL from the start. This is not currently compatible with the proxy.

If you can find a way to disable the EnableSsl option in your client, you'll be able to use it with the proxy. Otherwise, one potential solution would be to extend the proxy to support local as well as remote STARTTLS. You'd need to provide your own certificate for this, but you're already doing that, so I'm assuming there wouldn't be an issue there.

I'm happy to look at adding this feature on a consultancy basis if it needs prioritising.

[gerneio]

Thanks for confirming my suspicions. I was hoping that I was just misconfiguring something, but like you mentioned, looks like the proxy wasn't designed for this particular connection flow. I will probably take a stab at trying to implement the Client --> Proxy STARTTLS and see what I can come up with.

[gerneio]

Well I ended up finding a workaround for my particular situation. Found that I could install IIS SMTP Server Relay and place that in-between my email client and the proxy. The relay properly supports the hardcoded EnableSSL=True property set by the .NET email client (once certificates are configured under the relay), which then forwards the connection appropriately to the proxy (w/o certificate configuration). So my current flow is:

.NET SmtpClient --> IIS SMTP Relay --> Email Proxy --> Email Provider (Office 365)

Ultimately, it would be nice to get the proxy configured to support bi-directional STARTTLS flows to avoid this MITM relay, but this gives me something usable now so that I can move on with my actual use case, so I'll go ahead and close this issue for now.

Thanks for hearing me out!

[simonrob]

Thanks for following up - I'm glad you were able to find a workaround here.

I'll look into client STARTTLS at some point if I get chance.