Make SSL handshake failures diagnosable

[alexkeh]

We are testing Oracle's ADO.NET provider for .NET Core on macOS. I'm on the Oracle team myself and am looking for some direction on whether this is a .NET Core bug or something my team needs to change to support Apple SSL.

The provider hits a failure during SSL handshake. We have tested on Windows and Linux with TLS/SSL successfully with the same code.

We are using .NET Core 2.2.105, macOS 10.13.6, and TLS 1.2. The full stack trace is below.

Unhandled Exception: Oracle.ManagedDataAccess.Client.OracleException: ORA-00542: Failure during SSL handshake 
---> OracleInternal.Network.NetworkException: ORA-00542: Failure during SSL handshake 
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. 
---> Interop+AppleCrypto+SslException: handshake failure
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at OracleInternal.Network.TcpsTransportAdapter.Negotiate(ConnectionOption conOption)
   --- End of inner exception stack trace ---
   at OracleInternal.Network.OracleCommunication.DoConnect(String tnsDescriptor)
   at OracleInternal.ServiceObjects.OracleConnectionImpl.Connect(ConnectionString cs, Boolean bOpenEndUserSession, OracleConnection connRefForCriteria, String instanceName)
   --- End of inner exception stack trace ---
   at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPwd, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch)
   at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassword, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch)
   at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, SecureString securedPassword, SecureString securedProxyPassword, OracleConnection connRefForCriteria)
   at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
   at ODPC.Program.Main(String[] args) in /<directory location>/Program.cs:line 12

[stephentoub]

cc: @bartonjs

@alexkeh, to help with more info, could you try with .NET Core 3 Preview 5? It might be instructive to see whether it still repros.

[alexkeh]

@stephentoub
We don't have an ADO.NET provider working on .NET Core 3 yet, which makes us unable to try SSL on it.

[stephentoub]

@alexkeh, thanks, though that's interesting in and of itself. Your library that runs on a previous .NET Core version doesn't "just work" on 3.0? What fails?

[alexkeh]

@stephentoub
My team tried .NET Core SDK Version: 3.0.100-preview5-011568, which resulted in the same error:

Unhandled Exception: Oracle.ManagedDataAccess.Client.OracleException: ORA-00542: Failure during SSL handshake ---> OracleInternal.Network.NetworkException: ORA-00542: Failure during SSL handshake ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+AppleCrypto+SslException: handshake failure

I spoke too soon about not being able to run our library on the .NET Core 3.0. Since we support .NET Standard 2.1, we can run on .NET Core 3.0 after all. Nonetheless, we see the same error as when running on .NET Core 2.x.

[wfurt]

what exact version of 2.x do you use @alexkeh ?
Some ssl fixes were back-ported to 2.x so make use you use latest version.

cc: @bartonjs

[alexkeh]

@wfurt
We're using .NET Core SDK 2.2.105.

[wfurt]

I assume the server is not publicly reachable, right?
It may be handy to get packet capture. That at least should show server certificate.

[wfurt]

We will need more information @alexkeh how to reproduce it. Preferably sample code or specific steps.
Without it, this is not actionable.

BTW how did you establish trust between client and server?

[alexkeh]

@wfurt
The server is not publicly reachable. I created a MS case number 119052125003241 yesterday to enabling sharing of diagnostics and code that cannot be done in the public domain.

@ScotMac
Can you answer @wfurt's questions?

[ScotMac]

I believe we should be able to either produce a diag patch to get the SSL packets or do some type of packet sniffing (wireshark). Would that be sufficient?

In terms of the trust, i believe it is verisign trusted root, but i need to verify.

Though the server is not publicly reachable, @wfurt is it possible that we could get some get some type of more private communication channel open, that we could use to transfer the server connect details to you? eg, we could post one of our email addresses here, and you could initiate a conversation via that email address.

[wfurt]

wireshark may help. At least we would see certificate and part of the handshake.

The question is if this is ADO.NET specific or not.
Can you reproduce it with just ssl streams?
You can also try to connect with HttpClient or Safari.
It will fail on L7 protocol level but at least you would see if you can complete TLS handshake.

We can certainly set up alternative communication channel if you think that is useful.

[ScotMac]

The trust is explicitly setting a trustpoint for a digicert root in the cert of the server and client. Not using default CA's.

[ScotMac]

We are trying the connection establishment from our Oracle provided C client, sqlplus, to verify whether the problem is associated w/ .NET or the MacOS C libraries.

[ScotMac]

Looks like it is a Core issue. It doesn't happen for sqlplus (C client), and does happen on linux with a pretty similar stack.

Linux failure:

Unhandled Exception: Oracle.ManagedDataAccess.Client.OracleException: ORA-00542: Failure during SSL handshake ---> OracleInternal.Network.NetworkException: ORA-00542: Failure during SSL handshake ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
   at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
   at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
   at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
   at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
   at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet`1 downloaded, HashSet`1 systemTrusted, TimeSpan& remainingDownloadTime)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
   at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at OracleInternal.Network.TcpsTransportAdapter.Negotiate(ConnectionOption conOption)
   --- End of inner exception stack trace ---
   at OracleInternal.Network.OracleCommunication.DoConnect(String tnsDescriptor)
   at OracleInternal.ServiceObjects.OracleConnectionImpl.Connect(ConnectionString cs, Boolean bOpenEndUserSession, OracleConnection connRefForCriteria, String instanceName)
   --- End of inner exception stack trace ---
   at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPwd, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch)
   at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassword, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch)
   at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, SecureString securedPassword, SecureString securedProxyPassword, OracleConnection connRefForCriteria)
   at Oracle.ManagedDataAccess.Client.OracleConnection.Open()

[wfurt]

The information about certificate trust does not have enough details. There are several recent posts similar to this where the the issue is server certificate. Windows implementation seems to be more liberal but OpenSSL correct v3 flags are required.

You can try this (on Linux) to verify that your general trust is correct:

openssl s_client -connect HOST:PORT -showcerts

post output of that command @ScotMac if it connects successfully but SslStream does not.

[ScotMac]

@wfurt I believe our server requires two way auth. ie, the client will also need t authenticated. I am not that familiar w/ openssl, but it appears the s_client command take a -cert arg param. I believe the arg is a filename? Can the filename be a pkcs12 file?

[wfurt]

no, I don't think it can take pkcs12. https://www.openssl.org/docs/manmaster/man1/s_client.html
But you can extract certificate and key from pkcs12 using openssl pkcs12
You can also look at examples here: https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html

[wfurt]

BTW I also noticed that the new stack fails on LoadMachineStores(). Since you mentioned pkcs12: did you import it via X509Certificate2.Import? As I mentioned before, you can share simple SsslStream example of your code without any private ADO.NET as well as describe all steps you did to establish trust @ScotMac

[ScotMac]

Yes, i looked at exactly that manpage before i send the last message. It has no details of what the -cert takes. But i found here a good description of how to convert the pkcs12 to something the s_client will take here:

https://techdrawer.wordpress.com/2014/10/07/testing-ssl-with-your-certificate-using-s_client/

Specifically, i will try the following:

1. The resulting pcks12 (.pfx, .p12) can be converted to PEM format
   openssl pkcs12 -in <.p12 filename> -out -nodes
2. export the private key from the pkcs12 with the -nocerts flag
   openssl pkcs12 -in <.p12 filename> -out -nodes -nocerts
3. test with openssl s_client
   openssl s_client -connect host:port -cert <cert.pem> -key <key.pem> [-CApath ]

[bartonjs]

@ScotMac for your Linux on:

Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.StorePal.LoadMachineStores()

Seems like there's a file in /etc/ssl/certs/ (or wherever the system trust lives on your particular distro) that can't be opened by the current user.

[ScotMac]

Yes, the linux issue appears to be environmental, and we are now able to connect successfully from Linux.

However, the MacOS issue remains. Would you like me to try the openSSL command from MacOS?

[wfurt]

on MacOS we do not use OpenSSL for SslStream so it is not quite valid test. For client cert, I assume you use Import function as on Linux, right?

You never answer question about what certificate server uses and how do you set up trust. If that is certificate signed by well known CA, you should have it already in your KeyChain but it would be good to verify that.

You can do Wireshark trace from Linux (working) and OSX (notWorking) Looking at the difference may provide some clues.

[ScotMac]

Regardless whether you use OpenSSL on MacOS or not, i assumed you wanted to see the server cert via openssl. ie, that -showcerts option you indicated.

In terms of the trust, the certificate is digicert and both client and server are adding an EXPLICIT trustpoint for the digicert CA/root. We are not relying on "well known CAs" at all. Sorry if my previous response was not clear.

[wfurt]

ok. So you add it to System Root or System KeyChain on OSX?

[ScotMac]

No. Sorry if i wasn't clear again. We are adding that EXPLICIT trustpoint to the client and server certs, not the O/S.

[alexkeh]

I uploaded the test case to the MS Support case this morning. You can use to reproduce the issue and perform further diagnostics.

[karelz]

To our knowledge it turned out to be PFX that Mac OS didn't like.

We should have sufficient logging that enables us to diagnose these kind of problems easily - renaming the issue

[karelz]

Duplicate of #2359

[ScotMac]

This issue is associated w/ .NET Core SSL on MacOS failing to parse a perfectly valid pkcs12 file. ie has almost nothing to do w/ "exceptions and logging".

[wfurt]

The goal is to make such cases easier to diagnose.

[alexkeh]

@karelz I filed the original issue. You changed the issue title, immediately afterwards classified it as a duplicate, then closed it. I don't understand the purpose of doing all that.

The new title does not reflect the fundamental issue: .NET Core SSL on macOS is not working under certain conditions. I am not asking for improved logging to diagnose SSL handshake exceptions. And once this improved logging is added, will the macOS SSL .NET Core I identified be fixed (Issue #2359)? No.

Please re-open the issue and provide a title more reflective of the fundamental issue, such as the original one I used. @bartonjs has scheduled this issue to be fixed in .NET 5. I do not want him to get the mistaken impression that the issue no longer exists now that it is "closed" when it still does exist.

Thanks!

[bartonjs]

@alexkeh My portion of the work is dotnet/corefx#40539, and won't involve touching SslStream at all.

I presume @karelz's assessment is that once the "Apple doesn't find the private key from Oracle PFXes" is mitigated that the only thing left from this issue is that it was hard to find out where things fell apart... and that improving diagnosability was already tracked by somewhere else.

[karelz]

Correct. That is what my comment meant. I am curious which part was not clear.

Rejection of the PFX is done at OS level (not something .NET Core can influence). The remaining work is to make sure it is easier to debug when it happens next time -- which is tracked already by existing issue we discovered during triage and duplicated against.