Connections being forced closed

[RoyDonk]

We are seeing the below error pop up. At first it seemed to happen infrequently, then started increasing. Now some servers are completely unable to connect and get the error every time.

POP (localhost:1995; 127.0.0.1:64798->outlook.office365.com:995) Caught network error (server) - is there a network connection? Error type <class 'ConnectionResetError'> with message: [WinError 10054] An existing connection was forcibly closed by the remote host.

[simonrob]

I've not experienced this myself (though don't use POP personally). The only similar behaviour I've seen is Office 365's tendency to disconnect you temporarily if you are trying to download large numbers of emails in quick succession via IMAP (for example, a message similar to: Request is throttled. Suggested Backoff Time: [n] milliseconds).

If this is not the case, please could you provide more details: the OS, Python and proxy version, and the relevant sections from your configuration file.

[RoyDonk]

We are not using IMAP only POP and I think the email volume is fairly low. I have started to increase the interval between our mail server reaching out to see if that helps, but it doesn't seem like it has so far.

OS: Windows Server 2019 Standard. Build 17763
Python: 3.11.0 (One server is using 3.11.1 as a test to see if it made a difference)
Proxy: 12-14 release

Config:
Server Setup

[POP-1995]
server_address = outlook.office365.com
server_port = 995

Account setup

[[email protected]]
permission_url = https://login.microsoftonline.com/common/oauth2/v2.0\authorize
token_url = https://login.microsoftonline.com/common/oauth2/v2.0/token
oauth2_scope = https://https://outlook.office365.com/POP.AccessAsUser.All offline_access
redirect_uri = http://localhost
client_id = our client id
client_secret = our client secret

[emailproxy]
delete_account_token_on_password_error = True
encrypt_client_secret_on_first_use = False
allow_catch_all_accounts = False

We have 5 accounts on each server w/ 10 servers. All accounts are part of the same tenant.

[simonrob]

You've got a stray \ in your permission_url and duplicated https:// in your oauth2_scope. I'd try starting again from the example configuration file entry and see whether that makes a difference.

[RoyDonk]

I edited my previous comment with the actual text from the config file. Those mistakes are not actually in the config file. I just typed it out instead of using copy/paste from the remote server.

[simonrob]

In that case I suspect this is something to do with your server's TLS version – see Microsoft's help article. The proxy uses whatever is available, so perhaps your Python installation does not have recent enough TLS support (e.g., TLS 1.2).

Another issue was resolved by installing the certifi module, though I suspect this will not help in your case, and the issue is with the SSL library Python is using.

[RoyDonk]

I started using wireshark to take a look at the handshake and it does seem that the correct version of TLS is being used. Below are the screenshots of two different handshake attempts from the same server a few minutes apart.

Client Hello failure
Client Hello Success

One failed the other was successful. They both show that the client hello is being sent with TLS 1.2, I looked through the available ciphers and it includes TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 which is the preferred cipher for outlook.office365.com.

The only difference I see is the top line. The successful Client Hello shows "TLSv1.2 Record Layer: Handshake Protocol: Client Hello", while the failure shows "TLSv1" instead of TLSv1.2. From what I have read this is just the behavior of Wireshark when the server does not respond Client Hello.

I tried installing the certifi module as well, but it didn't help.

[simonrob]

Were you able to make any more progress in identifying the issue here? I noticed a similar sounding comment posted on Microsoft's most recent blog about the basic auth deprecation, and their reply has a link to resources related to the TLS 1.2 requirement and also a suggestion to enable security defaults if that helps.

[RoyDonk]

I haven't but it isn't doing it anymore. After I did the packet trace and saw the tls handshake seemed to be getting sent out properly I opened a ticket with Microsoft. They did their own trace and sent it to their network team. They keep saying they are looking into it and will let me know, but the issue stopped shortly after that. I'm not sure if it was a coincidence or if they just fixed something quietly on their end. I was testing some changes, but only on one server so it couldn't have been anything I did.

Thanks for your time and thanks for creating this, it really saved us.