Request support for more dynamic mTLS client cert/csr management

[dekimsey]

What would you like to have changed?

An interface to produce/configure the mTLS client CSR, or possibly allow the existing one to ignore it altogether.

Ideally, I'd like to be able to issue a custom CSR based on the information about the server and destination backend. Bonus points for being able to decide the details of the client cert during the TLS negotiation at the CertificateRequest phase (caddy server/module, server offered CAs, configured san, etc).

Why is this feature a useful, necessary, and/or important addition to this project?

Right now when using mTLS only the common name of the certificate can be set and it must be hard-coded (even replacers aren't supported). In a TLS negotiation a server may offer support for multiple CAs (which means different possible valid client certs we might chose from) or a client may want to use different client certs depending on destination backend. In effect, the reverse of a just-in-time server cert issuance by leveraging the CertificateRequest step of the TLS negotiation.

I am implementing Vault PKI issuance support in my application, but Vault is somewhat picky about the cert request details and I'd like to be able to more fully specify the details of the request dynamically (or just use the REST api pki/issue/$thing that gives me back a pkey instead of the sign api pki/sign/$thing which uses a CSR as an input). Unfortunately, Caddy/Certmagic's interface here don't let me manipulate (or outright ignore) the internally produced CSR.

What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

If I attempt to be sneaky about it (say by making my module also a KeyIssuer and then caching the key internally in a pubkey mapping) the cache storage engine gets confused. There might be a bug there in fact if the subject in the CSR when it's being stored != the subject that's actually in the cert. It attempts to read the wrong path on disk and fails. Regardless, it's a bit of a hack to take this approach.

Please link to any relevant issues, pull requests, or other discussions.

https://ldapwiki.com/wiki/TLS%20Full%20Handshake
https://www.vaultproject.io/api-docs/secret/pki#generate-certificate-and-key
https://www.vaultproject.io/api-docs/secret/pki#sign-certificate

BTW, thank you! The caddyserevr project's extensible design is really something amazing.

[mholt]

Thanks for the feature request! I'm, uh trying to decide if this is proper for CertMagic or for Caddy. I guess the API has to be exposed in CertMagic at least.

Can you be a little more specific about the use case / requirements? Because:

I'd like to be able to issue a custom CSR based on the information about the server and destination backend

I'm not sure I follow; what "server" do you mean and what is "destination backend"?

Bonus points for being able to decide the details of the client cert during the TLS negotiation at the CertificateRequest phase (caddy server/module, server offered CAs, configured san, etc).

So by this I think you are talking about hooking into the actual TLS handshake, right? "
"CertificateRequest phase" -- do you mean the part of the handshake that validates the server or client certificate?

[dekimsey]

Thanks, it seemed to cross the boundaries a bit too. But I wanted to start in certmagic, since it's the one doing the Key/CSR/Sign bits and getting hooks into there would help.

I'm not sure I follow; what "server" do you mean and what is "destination backend"?

Hrm, I'm sure I'm screwing up the terminology. But in my thinking both the contextual information such as the destination dial address, the sni, and the configured server module would be relevant.

So by this I think you are talking about hooking into the actual TLS handshake, right? "
"CertificateRequest phase" -- do you mean the part of the handshake that validates the server or client certificate?

Yes hooking into the server's ask for the client cert in the mutual authentication flow. In other words, when the server says "I need you to provide a client cert, here are the CA's I'll accept". The client selects and then responds with a cert. Right now, it's not possible for a module to hook into that.

[mholt]

@dekimsey Oh, are you talking about Caddy's reverse proxy module specifically?

[dekimsey]

Ah! Yes, I'm using TLS client auth with my backends with the reverse proxy and external l4 modules to wrap my connections for an mTLS service I have to communicate with.

[dekimsey]

That bit is definitely Caddy. But the underlying tooling to manage the client certs seemed certmagic's which is why I opened it here. I think I didn't do a very good job of disambiguating the two.

So the ask is intertwined I think.

1. Add Certmagic interface for advanced cert creation (CSR) (hopefully supporting more context about who's asking and where to?).
2. Extend Caddy's upstream TLS support to issue/select the appropriate client auth cert at the CertificateRequest phase of TLS.

[mholt]

Ok that makes way more sense. Thank you for clarifying!

[francislavoie]

Issuance of certs by Caddy's locally managed CAs happen in Caddy's PKI app (which calls out to Smallstep libs), and not in Certmagic https://github.com/caddyserver/caddy/tree/master/modules/caddypki

So I think this is actually entirely a "Caddy" thing.

[mholt]

@dekimsey Are you talking about CA certificates? Or server certificates issued by a CA?

[dekimsey]

I'm speaking of client certs the Caddy app would use to authenticate to backends.

In my use case, I have a local proxy adding client certs to backend connections. So that's the TLS client authentication that happens in

• apps › layer4 › servers › routes › handle › proxy › upstreams
• apps › http › servers › routes › handle › reverse_proxy

I tried a couple of iterations playing with caddytls.CertificateLoader, certmagic.CertManager, and certmagic.CertificateSelector but they don't seem to be involved in client auth negotiation. The best I could do was a certmagic.Issuer and a caddy.App (extraneous, handles the Vault auth session management)

Below is an example configuration of what I've been doing (with my own Vault-based Issuer and App modules). I've pruned a bit but its working even if the client cert details must be hard-coded.

{
  "apps": {
    "http": {
      "servers": {
        "foo": {
          "listen": [
            ":8000"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "client_certificate_automate": "[email protected]",
                      "server_name": "http-foo-service",
                    }
                  },
                  "upstreams": [
                    {
                      "dial": "mtls.example.com:443"
                    }
                  ]
                }
              ]
            }
          ],
          "automatic_https": {
            "disable": true
          }
        }
      }
    },
    "layer4": {
      "servers": {
        "bar": {
          "listen": [
            "unix//tmp/bar.sock"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "mtls.example.com:443"
                      ],
                      "tls": {
                        "client_certificate_automate": "[email protected]",
                        "server_name": "tcp-bar-service"
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "vault",  // certmagic.Issuer module
                "role_path": "pki/sign/a-role-name",
                "ttl": 3600000000000
              }
            ],
            "disable_ocsp_stapling": true
          }
        ],
        "renew_interval": 1000000000
      }
    },
    "vault": {  // caddy.Vault
      "address": "https://vault.example.com",
      "auto_shutdown": true,
      "auto_renew": true
    }
  }
}

[mholt]

Thanks, I'll revisit this soon -- currently trying to get Caddy 2.6 into beta

[dekimsey]

Thanks @mohlt, no worries. I've spent some time thinking about what I was doing and wanted to try to share what I did try and the issues with it.

Here's what I gathered in trying to generate client certs for my upstream connections

Between certmagic.KeyGenerator and certmagic.Issuer there exists no way¹ to manipulate certmagic's CSR say to add additional SAN's, customize the OU, alter the subject name entirely, or add extended fields.

The certmagic.Manager is only be invoked for the TLS clientHello phase. I believe this is a design issue with the certmagic interface here. There is no method/interface for the CertificateRequest step of the TLS negotiation (if caddy even had a way to hook to it). In other words, I found it better to think of GetCertificate as OfferServerCertificate. Thus making it clear there is no corresponding OfferClientCertificate.

I did fiddle with a caddytls.CertificateLoader, but it doesn't have any mechanism to ask again or stream certs (say load certs from a channel) so I cannot figure out anyway to update certs after the initial load. If there is a way for the module to ask for a re-load, I couldn't identify one.

The client_certificate_automate triggers a cert request in the config immediately, I had hoped some of the above methods would let me delay or customize the client cert being issued but I didn't have any success. Since this field must be the subject name of the cert, there's no way really to issue or configure multiple more advanced issuers.

Footnotes

1. I can sorta hack support by having a module perform the certmagic.Issuer and certmagic.KeyGenerator. Since the module then knows the private key, it can ignore the given CSR and generate one anew. However it exposes a bug in the on-disk cert storage where it's unable to read the file after writing it (the expected CSR's subject name differs from the actual cert subject name). I really ought to try to construct and submit a test for it. I've seen a public CA re-order the san/subject names in a CSR, so I know that's a things CAs may do. ↩

[mholt]

Between certmagic.KeyGenerator and certmagic.Issuer there exists no way1 to manipulate certmagic's CSR say to add additional SAN's, customize the OU, alter the subject name entirely, or add extended fields.

It's true, for managed certificates, CertMagic only manages single-SAN certificates so adding SANs is not possible (it would require significant refactoring and complexity and goes against best practices). If you want to just get a certificate (via ACME, at least) and not worry about managing it (because CM requires single-SAN) then you can drop a level lower and use acmez directly which does give you control over the CSR.

As for extended fields, maybe we can look into a way to open up an API for that. Do you have a suggestion? Maybe a new field in the Config struct?