dev-certificates on Linux -- how to get dotnet-to-dotnet comms to work? #7246
Comments
|
If it would be helpful, I can possibly set up a self-hosting IdentityServer project at some point, given a little time. I can't supply the full code I'm working on because it's for a client. |
Eilon
added
the
area-commandlinetools
|
@fluffynuts Were you able to resolve this in any way? |
|
@maq1823 no, I didn't )': I've read that this is an issue within dotnet itself (specifically to do with cert trusting on linux), but I don't know any more about it. I eventually had to stop development under linux and boot into windows as this was a show-stopper preventing any progress on the project. |
|
I'm having the same issue - I can find a lot of details for how to enable host-to-container SSL trust, but not for container-to-container trust. |
|
Facing the same issue. I am trying to setup ASP.NET Core 3.0 Web Api, with self-hosted OpenIddict middleware, for handling OAuth token generation. In Windows, things work fine, even with dotnet generated TLS certificate. There is no issue while trying to access the application URL, Token URL etc. This is blocking development/deployment of my project in Linux environment. Would really appreciate some one looking/responding to it. |
|
Is this about development time or when deploying for production? The dev certs probably shouldn't be used for production purposes. |
|
@poke development time. Unless the use cases work we can't be even thinking about prod. |
|
https://github.com/dotnet/corefx/issues/40725 You need to trust it in libcurl I believe. Regardless, it's a corefx issue not an aspnet one. |
|
@blowdart I retried by asking libcurl to trust the self signed certificate: But the issue still happens: The problem does not seem to related to libcUrl though, since the following works fine: The problem happens when JWT middleware internally tries to fetch config from OAuth endpoint, by making HTTPS call to: But somehow that does not stop JWT middleware to throw the error. |
|
@terrajobst this appears to be a corefx problem rather than aspnet core. Can someone explain how to get corefx to trust a self signed cert? |
|
I don't understand what the repro is exactly here. Can you attach the dev certificate so we can look at it?
The problem is most likely not a "trust" issue. We have see that installing self-signed certificates into the trust list on Linux doesn't work because the certificate itself is usually missing some X509v3 attributes that Linux (due to OpenSssl) require but Windows doesn't. OpenSsl on Linux requires that self-signed certificates have the "Cert Signing" attribute on them. This is because a self-signed, trusted certificate has to be both a valid CA certificate (requires 'Cert Signing') as well as a "leaf node" certificate. |
|
@davidsh please find attached the dev certificate file with which the issue happens. It would also be very helpful if you could point to any article where the proper step for generating self-signed certificates are provided, that would make linux/openssl happy. |
|
I have this issue too. But for me is issue: |
So, I've researched this and determined that on Linux, self-signed certificates will not work unless you have "Certificate Signing" as an additional attribute on the KeyUsage field.
This limitation is not a .NET Core thing. It is an OpenSsl change (between 1.0.2e and 1.0.2g) where self-signed certificates need to have this additional attribute set on them. Even when you install the certificate (public key portion) using 'update-ca-certificates' on Linux, the certificate will still not be "trusted" if the "Certificate Signing" bit is missing. There is an open issue still on the OpenSsl repo about this problem: openssl/openssl#1418 This is a client-side problem. For example, I created an ASP.NET Core website using kestrel and hosted it on my Windows box. I first used the default "ASP.NET Core HTTPS development certificate" for the website. I then installed the certificate (public key portion) into the trusted certificates list on my Linux machine using 'update-ca-certificates'. When I used 'curl' to send a request to the website, I got an error. When I changed the website to use a self-signed certificate that has the added "Certificate Signing" attribute, then 'curl' worked from the Linux machine. I got the same results using HttpClient from the Linux machine. I got errors about the certificate when I used the default ASP.NET Core development certificate. But when I updated the certificate to include "Certificate Signing" then HttpClient worked again. As a result of this investigation, we need to fix the 'dotnet dev-certs' tool so that it will generate a self-signed certificate that can be used on both Windows and Linux properly. @Tratcher where is the source code to this tool? Perhaps I can submit a PR to fix that. Instead of using the default ASP.NET development certificate, you can generate your own self-signed certificate using Windows PowerShell or Linux OpenSsl tools. Just make sure to specify the 'Certificate Signing' attribute in addition to other things. For example, this is the command I used to generate a self-signed certificate on Windows. New-SelfSignedCertificate -DnsName "localhost" -KeyUsage CertSign, DigitalSignature, KeyEncipherment -FriendlyName "Better ASP.NET Core HTTPS development certificate" -CertStoreLocation "Cert:\CurrentUser\My"I then exported the public key portion and installed that on my Linux machine and ran 'update-ca-certificates' to put it into the trusted list. |
|
cc: @wfurt |
|
As @davidsh mentioned, OpenSSL is more picky about correct flags. If curl does not work without There is longer discussion about this here: https://github.com/dotnet/corefx/issues/37516 my suggestions would be to clear old certificate, create new set, verify v3 extensions and verify that curl can connect without -k. |
As I discussed above, the current 'dotnet dev-certs' tool is generating a self-signed certificate that works only on Windows but can't work on Linux due to OpenSsl requirements that self-signed certificates have the additional 'Certificate Signing' bit set on them. So, "clear old certificate, create new set" will not work at all unless that new set contains the additional 'Certificate Signing' bit. And as you pointed out, the current ASP.NET 'dotnet dev-certs' tool is not creating a certificate that can make Linux clients happy. That Key Usage attribute is missing the 'Certificate Signing' bit. So, the best fix for everyone is to 1) understand the problems that OpenSsl on Linux has regarding self-signed certificates, and 2) change our 'dotnet dev-certs' tool so that the automatically generated ASP.NET Core development certificate has the proper attributes so that it works on both Windows and Linux. That will reduce pain for developers. cc: @bartonjs |
|
If this were to change it should only be on Linux. Creating a trusted cert which can issue other certs goes beyond self signed, it's creating a root CA and that is dangerous. I'd argue OpenSSL isn't being correct here, but it's easier for us to cope with their bad security decision than it is to get it rolled back, but if we do so we don't want to expose Windows users to additional risks just because of OpenSSL. As for Linux to Windows communication, the dev certificate is limited to localhost anyway, so it should be rejected for cross machine communication, unless you're disabling subject validation which is not something we want to encourage at all, so I view that as out of scope and where folks need to start considering their own development certificate infrastructure and creation. |
What's more interesting is that the certificate itself has a Basic Constraint that says it isn't a CA but an end-entity certificate.
It's just that OpenSsl has some logic that says any self-signed certificate can't be "trusted" unless it has the "Certificate Signing" attribute in the KeyUsages field. I'm going to explore other possibilities for creating the self-signed certificate to see what works on both Windows and Linux. Perhaps if the Basic Constraints were changed from to then it would further reduce risk if someone tried to use this development certificate to issue further certificates from it. |
|
@davidsh thanks for your reply. My development platform is linux, so I used openssl to do the same. I transferred the crt file to windows machine, and checked the KeyUsage field, and Certificate Sigining was indeed added.
However, the issue continues to happen. Same exception as I mentioned in my preview comments. Can there be any other attribute missing? |
How did you generate the self-signed certificate exactly? What was the openssl command you used? What version of openssl do you have? Is the self-signed certificate (*.PFX file with private key) installed on the Linux machine? How/where did you install that? Is the Linux machine the server machine? Is the self-signed certificate public key portion (*.CRT file) installed in the machine trusted CA store with commands similar to? sudo cp ~/dev-cert.crt /usr/local/share/ca-certificates/dev-cert.crt
sudo update-ca-certificates |
I used the following commands (localhost.conf commented above) I am using OpenSSL version 1.1.1
Yes, using the commands above.
Yes.
Not installed but I call curl like this: |
|
Is this an ASP.NET Core server? How is the ASP.NET Core Server finding the self-signed certificate (w/ private key)? I've never tried your particular set of instructions, i.e. using 'certutil' on the Linux machine. What happens if a Windows machine tries to access this Linux http://localhost:44320 server? And assume that you have installed the self-signed certificate (public key portion) into the Windows trusted root store. Will that work without any browser errors? |
|
Yes, it is a ASP.NET Core server. Above code shows how the server finds the certificate. Also, as I mentioned above, I call another api endpoint to get access token, and then, as I make an api call to access a protected end point, |
|
Using |
|
So, I did a test where I created a sef-signed certificate on Windows using PowerShell. This self-signed certificate has the 'Certificate Signing' attribute on it. PS C:\Users\davidsh> New-SelfSignedCertificate -DnsName "localhost" -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.1","2.5.29.19= {critical}{text}false") -KeyUsage CertSign, DigitalSignature, KeyEncipherment -FriendlyName "V2 ASP.NET Core HTTPS development certificate" -CertStoreLocation "Cert:\CurrentUser\My"I then ran an ASP.NET Core web server on Windows using this self-signed certificate. I was able to access it from a Linux WSL window using 'curl' and making sure the self-signed certificate was trusted (via passing the .crt via a parameter to curl). davidsh@davidsh2:/mnt/d/dotnet/web1$ curl --cacert ./localhost-v2.crt --verbose https://localhost:5001/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: ./localhost-v2.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=localhost
* start date: Oct 11 17:27:55 2019 GMT
* expire date: Oct 11 17:47:55 2020 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: CN=localhost
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffbe609b40)
> GET / HTTP/2
> Host: localhost:5001
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Fri, 11 Oct 2019 17:44:35 GMT
< server: Kestrel
<
* Connection #0 to host localhost left intact
Hello World!davidsh@davidsh2:/mnt/d/dotnet/web1$I will try your openssl instructions to generate the self-signed certificate on Linux itself. And then try to run the Linux server and use curl against it. But our respective repros are not actually the same because you have a Linux solution where you have the ASP.NET server do an outbound request back to itself over TLS. Can you post a simple complete repro for that? We have too many variables at work here and we need to get a standardized repro. |
|
@BorisWilhelms just out of curiosity, do you run the script in the project dir? I am running my project in a remote env and relying on port forwarding. Just can't seem to get it to work. I'm on ubuntu 20.04, dotnet 5. Thank you. |
|
@nickwang0808 no. The script is not made for remote environments. The script is for local development machines where you run the browser and the server on the same machine. Could you please explain your use case more? |
|
@BorisWilhelms I am running on vscode with the remote development extension pack, the host is ubuntu 20.04 and the client is running on windows 10. the way microsoft doc explains is that vscode setup automatic port-forwarding and I can just go to localhost:5001 to access the webpage built by .net. it does seems like the certificate is there but chrome is not taking it? One thing I noticed is that if I remove the certificate or just run the Again really appreciate you time helping. |
|
@nickwang0808 Are you using Devcontainer in VS Code? If so please check your devcontainer.json. The process to reuse your trusted development certificate is described there. |
|
@BorisWilhelms not I am not, just a normal vm. |
|
For folks interested in this. You can get this to work on Ubuntu following these instructions Similar steps can likely be taken for other distros. |
|
@BorisWilhelms Thanks for your comment on stack overflow! This led me into the right direction regarding the 1.3.6.1.4.1.311.84.1.1 property needed on the cert. As I faced the issue that the cert was either accepted in Chrome or in Firefox but not in both dependent on CA:TRUE/FALSE. I created a script, which sets up a Trust Chain in a way that both browsers are happy. Root CA and signed server certificate. |
|
I'm having issues with SSL invalid connection between service requests in WSL as well. In my case I get the following output:
The above error occurs during access token validation when in my API tries to access the Identity service. I'm using the following powershell script to generate the certificate: which I then use to generate a I'm fairly new to certificates so I'm very likely missing something and I appreciate if you could point me towards any resource that could serve as guideline for the setup I'm trying to do. |
Good day! I am trying to run your script. I got the message: ./scripts/ubuntu-create-dotnet-devcert.sh: 2:.: Can't open ./common.sh. Please tell me how to solve the problem. |
|
This worked for me nicely:
I'm on Ubuntu 20.04. I was having trouble getting Identity authentication to work on our project, that said the docs really need to be updated because I spent a ridiculous amount of time not understanding what I was doing wrong, in fact I still can't get Firefox on Ubuntu 20.04 to fully trust the certs (the top-left lock still shows an exclamation mark but the authentication now works, Chrome seems to be fine). Here are the links to the docs: There should at least be a note on there that says there is known issue with |
|
There is "Edit" button on the doc pages. You can edit and submit PR with proposed changes @PhilParisot. Tech writers will then help you to polish the wording end details. |
|
Would they allow me to put on a note on there saying this is broken at the moment on Ubuntu 20.04? I haven't tested all distros, and I literally used a script someone else wrote to fix my issue, I don't know what the guidelines are but I'm pretty sure I'd get my PR rejected... |
|
I don't know. distro specific details are easy to get obsolete IMHO. |
|
Adding yet another small repro. The client code is out of our control, it's a callback into our service. Usual caveats about doing this in production -- it's not a production use-case, it's purely a developer use case, but it's blocking our linux-based developers from using HTTPS. Server: using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace service
{
public class Program
{
public static void Main(string[] args)
{
WebHost.CreateDefaultBuilder(args)
.Configure(applicationBuilder =>
{
applicationBuilder.UseRouting();
applicationBuilder.UseEndpoints(endpoints =>
endpoints.MapGet("/test", async context => await context.Response.WriteAsync("OK")));
})
.Build()
.Run();
}
}
}Client using System;
using System.Net.Http;
using System.Threading.Tasks;
namespace client
{
class Program
{
static async Task Main(string[] args)
{
var httpClient = new HttpClient();
var response = await httpClient.GetAsync("https://localhost:5001/test");
Console.WriteLine(response.StatusCode);
Console.WriteLine(response.Content);
}
}
}Output from % curl https://localhost:5001/test --verbose
* Trying 127.0.0.1:5001...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Developer CA; OU=Developers; CN=Developer self-signed Server Certificate
* start date: Nov 3 02:49:29 2021 GMT
* expire date: Nov 3 02:49:29 2023 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: O=Developer CA; OU=Developers; CN=Developer self-signed Root CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e197e74850)
> GET /test HTTP/2
> Host: localhost:5001
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Wed, 03 Nov 2021 16:05:06 GMT
< server: Kestrel
<
* Connection #0 to host localhost left intact
OK%And client output. % dotnet run
Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: PartialChain
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at client.Program.Main(String[] args) in /home/michaeljon/src/repro/client/Program.cs:line 12
at client.Program.<Main>(String[] args) |
|
Where did you put your |
|
It's impossible to work with .net Core on Linux! |
Why not? The claim does not have enough details. While there is no automatic creation and trust for dev certificate, you can use user distro tools to create one - just like you would for production. |
|
I have to concur with @shpsyte , Linux does feel like a second class citizen for .NET and it's frustrating, I still haven't been able to make my certs work on Firefox for Ubuntu |
|
@PhilParisot curious if this works for you? https://dev.to/lmillucci/firefox-installing-self-signed-certificate-on-ubuntu-4f11 |
No, it is not. I am doing it daily. But unfortunately, it is not that easy to support Linux, simply because there is no Linux. There are distributions and different distros have different cert stores. Therefore, applications tend to bring their own cert store. If you install the application as a snap package, the application cert store is in a different place, etc.... But if you know your distro and applications, it is possible to setup the certs for your workflow in a few minutes. Written from Microsoft Edge, running on Arch Linux, with .NET 3.1, NET 5 and .NET 6, Rider, Postman, Azure Data Studio. |
@DamianEdwards THANK YOU, yes it finally works! You have no idea how long I've searched for a solution! I'd REALLY appreciate if someone could update the docs here: https://docs.microsoft.com/en-us/aspnet/core/security/enforcing-ssl?view=aspnetcore-6.0&tabs=visual-studio#trust-the-certificate-with-firefox-on-linux with the solution. Creating the |
Very well said, just skip using donet dev-certs. Have a dev cert and avoid headaches. Would be helpful if the documentations that refer to building multi container or docker orchestration applications, kept things simple. |
Worked for me on Arch with .NET 6 and ASP.NET Core, ty! |
Could you be more specific? I'm facing the same issue. |
and others
Description
I have an asp.net core application with a self-hosted IdentityServer component. I can convince a browser (Chrome and Firefox) to visit the site, but I'm having trouble getting the IdentityServer component to be able to talk back to localhost due to certificate errors, specifically:
I realise that certificate installation can differ from distro to distro, however, I've tried this first on my (more native) Gentoo, and also in an Ubuntu vm:
dotnet dev-certs https -ep /tmp/dnc.pfx -p SomePasswordopenssl pkcs12 -in /tmp/dnc.pfx -clcerts -nokeys -out /tmp/dnc.crt3.5 in the case of Gentoo, I was also able to use certutil to install the crt into my ~/.pki/nssdb; in the case of Ubuntu, this dir does not exist and I'm not sure how to bootstrap it
To Reproduce
Set up an asp.net core project with self-hosted IdentityServer on Linux, such that the config for identity server points back to the same hosting application, using dev certs. Any route requiring authorization will result in a 500 error with the above logging in place. Using
dotnet dev-certs https --truston Windows, same code, no problem.Expected behavior
I expect to be able to get dotnet-to-dotnet comms working with the dev cert. I'm probably missing something )':
Ubuntu dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a
Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/
Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62
.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
Gentoo dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a
Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/
Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62
.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
I have also tried with dotnet binaries from https://dotnet.microsoft.com/download/thank-you/dotnet-sdk-2.2.103-linux-x64-binaries
The text was updated successfully, but these errors were encountered: