HTTPStatus.BAD_REQUEST Bad request syntax with Office365

[ilpincy]

When trying to access Office365 SMTP, I get the following error:

HTTPStatus.BAD_REQUEST Bad request syntax ('\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03\x84+ê\x15\x16¿pñ\x0fE¨6p\x0e¡é-?\x0faØ\x07[¦õK~')

If I try to decode the string on the command line, I get:

echo -n '\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03\x84+ê\x15\x16¿pñ\x0fE¨6p\x0e¡é-?\x0faØ\x07[¦õK~' | base64 --decode
Invalid character in input stream.

My configuration follows the examples and is as follows:

[SMTP-587]
server_address = smtp.office365.com
server_port = 587
starttls = True
local_address = [MY PUBLIC IP]
local_certificate_path = /etc/letsencrypt/live/[MY SERVER]/fullchain.pem
local_key_path = /etc/letsencrypt/live/[MY SERVER]/privkey.pem

[MY EMAIL ACCOUNT]
permission_url = https://login.microsoftonline.com/[MY TENANT ID]/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/[MY TENANT ID]/oauth2/v2.0/token
oauth2_scope = https://outlook.office365.com/SMTP.Send
redirect_uri = https://[MY PUBLIC SERVER]:18080
client_id = [MY CLIENT ID]
client_secret = [MY CLIENT SECRET]

I run the proxy as follows:

python3 emailproxy.py --local-server-auth --no-gui --debug

My goal is to send emails from GMail using its SMTP interface for aliases, so I am conducting my testing directly from there.

Thank you in advance for any help you can provide, and thanks for this great piece of software!

[simonrob]

I tried just now but wasn't able to replicate this issue. I had to select SSL rather than TLS in the Gmail alias settings because for whatever reason Gmail's TLS handshake with the proxy using my server's Let's Encrypt certificate did not work. However, after doing this it all works without issue, both with an imported pre-authenticated configuration file and via the --local-server-auth proxy option. Interestingly, if the proxy is stopped mid-way through the authentication sequence, Gmail incorrectly detects this as a successful login, but that's a different issue on their end.

In case it helps, I used a VPS (Debian 11) on which I've previously enabled TLS 1.0 via, e.g., sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1.0/' /etc/ssl/openssl.cnf (though as noted above I used SSL so this is probably not relevant). I used the latest version of the proxy for this test.

My configuration file was mostly the same as yours, but I don't have a configuration that requires a tenant ID so couldn't test that part. I'd be quite surprised if this is the issue, though (unless it is something to do with your client ID requiring administrator approval?).

Where are you getting this error, and is there any more context you can provide?

[ilpincy]

Thanks for the quick reply!

had to select SSL rather than TLS in the Gmail alias settings because for whatever reason Gmail's TLS handshake with the proxy using my server's Let's Encrypt certificate did not work.

I had to do the same. I also use a server with Let's Encrypt.

In case it helps, I used a VPS (Debian 11) on which I've previously enabled TLS 1.0 via, e.g., sed -i 's/MinProtocol = TLSv1.2/MinProtocol = TLSv1.0/' /etc/ssl/openssl.cnf (though as noted above I used SSL so this is probably not relevant). I used the latest version of the proxy for this test.

In my /etc/ssl/openssl.cnf I have no line enabling TLS.

My configuration file was mostly the same as yours, but I don't have a configuration that requires a tenant ID so couldn't test that part. I'd be quite surprised if this is the issue, though (unless it is something to do with your client ID requiring administrator approval?).

In my case I am trying to enable a university account, what you suggest might be the case. I'll reach out to our IT services.

Where are you getting this error, and is there any more context you can provide?

I see the message "Please visit the following URL to authenticate account"; I click on the URL, and I get redirected to the Microsoft login page. Then, I get another page that asks me whether I want to allow the SMTP.Send scope. Both pages are cleared with no issues. Then I get redirected back to the web server of the proxy, and that's when the error appears. Using logs, I could gather than get_oauth2_authorisation_code() is called, but none of the *_token() functions is ever called.

[simonrob]

Please could you share the log file with me? Feel free to email if you'd rather not post it here.

[simonrob]

I noticed that you are using an O365 oauth2_scope value that does not include offline_access. If this string is not present, a refresh token will not be provided. You will also be asked to manually authenticate far more often in this case.

Currently the proxy is set up to always expect a refresh token, so this could be one reason for the error (though doesn't explain the corrupted error string you encountered. 803cdce switches the code retrieval logic around slightly to only check the refresh token after trying the access token. Could you try this branch?

[ilpincy]

I added offline and used the branch you suggested. I still get an error, but the the output is different. The "syntax" part seems solved, although an error persists. I am certain that the client id, client secret, and tenant id are correct.

2022-09-02 17:59:35: *** get_oauth2_authorisation_code
2022-09-02 17:59:35: Authorisation request received for [MY ACCOUNT] (local server auth mode)
2022-09-02 17:59:35: Email OAuth 2.0 Proxy Local server auth mode: please authorise a request for account [MY ACCOUNT]
2022-09-02 17:59:35: Local server auth mode ([MY PUBLIC ADDRESS]:18080): starting server to listen for authentication response
2022-09-02 17:59:35: Please visit the following URL to authenticate account [MY ACCOUNT]: https://login.microsoftonline.com/[MY TENANT ID]/oauth2/v2.0/authorize?client_id=[MY CLIENT ID]&redirect_uri=https://[MY PUBLIC ADDRESS]%3A18080&scope=https%3A%2F%2Foutlook.office365.com%2FSMTP.Send%20offline&response_type=code&access_type=offline&login_hint=[MY ACCOUNT]
2022-09-02 17:59:38: Local server auth mode ([MY PUBLIC ADDRESS]:18080): received authentication response HTTPStatus.BAD_REQUEST Bad request version ('\x1a\x1a\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x93')
2022-09-02 17:59:38: Local server auth mode ([MY PUBLIC ADDRESS]:18080): received authentication response ü!u;jÒÒ[8v
æå"wP ðúÀ[îLuÍÉ�Í¿þuÂ�ì�ç�6VJÕÒÿ À+À/À,À0̨̩ÀÀ/5 400 -
2022-09-02 17:59:38: Local server auth mode ([MY PUBLIC ADDRESS]:18080): closing local server and returning response
2022-09-02 17:59:38: Authentication request failed or expired for account [MY ACCOUNT] - aborting login
2022-09-02 17:59:38: SMTP ([MY PUBLIC ADDRESS]:587; 209.85.210.52:39521->smtp.office365.com:587) <-- b'535 5.7.8  Authentication credentials invalid. Email OAuth 2.0 Proxy: Login failed - the authentication request expired or was cancelled for account [MY ACCOUNT]\r\n'

[simonrob]

Did you add offline or offline_access?

See the sample configuration file for an example of the correct scope string. What happens if you use that one?

[ilpincy]

offline_access, and that's the output I got. I tried with both and the output is the same.

[simonrob]

Ok, it's just that the log extract is offline, which is incorrect.

I also see that your redirect_uri is https. This is rarely (never?) the case for this sort of local usage. The proxy doesn't support https for this because it is always (currently) done on the same machine as the proxy. What happens if you switch to http here?

In addition, in almost all cases, the redirect_uri value is http://localhost, but because you've anonymised it to [MY PUBLIC ADDRESS] I suspect that you've used your server's local_address value here, perhaps mistakenly? Unless you've set up your O365 client ID/secret to explicitly use this address for the redirect, it will fail.

[ilpincy]

I also see that your redirect_uri is https. [...] What happens if you switch to http here?

Azure doesn't accept it. It says Must start with "HTTPS" or "http://localhost"

In addition, in almost all cases, the redirect_uri value is http://localhost, but because you've anonymised it to [MY PUBLIC ADDRESS] I suspect that you've used your server's local_address value here, perhaps mistakenly? Unless you've set up your O365 client ID/secret to explicitly use this address for the redirect, it will fail.

I used my public address, the one you can ping from the open web. I set the redirect_uri in the app registration on Azure, and the redirect is performed correctly. Does it make sense to use

redirect_uri = http://localhost
redirect_listen_address = http://[MY PUBLIC IP]:18080

Now a new issue popped up: when I try to change the app registration on Azure or create a new one, I get There was an error loading this content. Please refresh the page to try again. This is a problem with the Azure backend which I'll hopefully get solved after Labor Day...

[simonrob]

As the Azure message says, http://localhost is the correct value, and is needed to allow the proxy to intercept the request. Your public address isn't needed here - any public-facing use of the proxy is set up in the server configuration, rather than the account. redirect_listen_address is only needed if you need to map another address to the redirect_uri - see #42 for further discussion about this.

[ilpincy]

Alright, thanks! I'll get the Azure issue solved and try with http://localhost.

[simonrob]

ba3db1d makes this misconfiguration a bit more visible, giving a warning rather than the bad request error. The reason for requiring http://localhost is that it is a direct route to reconfigure the proxy, so you don't want it to be publicly accessible (and can't get a non-self-signed certificate for this address, so it must be http). Publicly-available redirect_uri values are used when you're giving others (e.g., web services, etc) access to your account. I'm happy to update/improve the documentation further to make this clearer.

[ilpincy]

Still no luck, but I am a little confused. I tried with the following configurations:

redirect_uri = http://localhost:18080

and

redirect_uri = http://localhost:18080
redirect_listen_address = http://[MY PUBLIC IP]:18080

None works: the browser complains that it can't open the server localhost:18080.

This is to be expected, and hence my confusion. I have no server running on my laptop (where localhost would point to); I run email-oauth2-proxy on a remote Linode machine reachable with [MY PUBLIC IP]. Azure does not allow me to set http://[MY PUBLIC IP]:18080 as a redirect URI; but I can set the https version, which works for redirection but then generates the error I reported.

I am not sure how to continue at this point. I do admit my understanding of OAuth2 and the Microsoft backend are minimal, so I'm sure there's something I am missing.

EDIT: Because what fails is the first authentication, I will try with an ssh tunnel from my laptop to the remote server. This should work, in theory, and allow me to set localhost as required.

EDIT2: The ssh tunnel did the trick, apparently! The final configuration was indeed with redirect_uri = localhost:18080 and no redirect_listen_address.

[ilpincy]

I'm going to close this issue since everything works as intended. Thank you so much for your help!

[simonrob]

Thanks for following up. Yes - if the proxy and client are separated, visiting the authentication redirection link needs to take place on the server rather than the client.

If you're interested, #33 discusses various ways to separate this a bit more, but currently, since you need to have access to the server to get the authentication link, it's typically okay to handle it on there.

I'm more than happy to look at pull requests that would develop this a bit more, but for the reasons mentioned in #33 it's not as trivial as it might first seem.

[ilpincy]

Thanks for the link, I learned a lot from reading that issue. Because this is intended as a personal solution, I think that the current setup is acceptable for my purposes as is.