helm: Add support for mounting existing TLS secrets with optional root CA

[webvictim]

This PR adds a tls.existingSecretName value to the teleport-cluster Helm chart which makes it easier to provide a custom TLS keypair for Teleport to use when securing its web UI while still using other chart functionality.

It also provides a tls.existingSecretNameCA value to mount a root/intermediate CA or other bundle into the pod using the SSL_CERT_FILE environment variable so that Teleport can validate the full certificate chain itself and prevent User Message: unable to verify HTTPS certificate chain errors in the pod logs.

Docs counterpart is #11306 (separate PR due to some local workflow issues, fixed now)

Fixes #11103

[mattmelgard]

Ran into some issues when deploying/testing this out this evening, part of which may or may not just be something missing on my end. Specifically -

It seems the image quay.io/gravitational/teleport:10.0.0-dev either doesn't seem to exist or I don't have permission to pull it since I was getting image pull errors when deploying the chart from your branch:

  Normal   BackOff      1s (x2 over 29s)   kubelet            Back-off pulling image "quay.io/gravitational/teleport:10.0.0-dev"

To get past this I overrode the value using teleportVersionOverride and set it to 9.0.0, but then encountered this error when I set the existing secret value to use my Cloudflare Origin certificate which was deployed as a k8s TLS secret in the cluster:

k logs teleport-86bf88644f-d7kf7
ERROR: "unable to verify HTTPS certificate chain in : \x1b[31mERROR: \x1b[0mWARNING:"

  The proxy you are connecting to has presented a certificate signed by a
  unknown authority. This is most likely due to either being presented
  with a self-signed certificate or the certificate was truly signed by an
  authority not known to the client.

  If you know the certificate is self-signed and would like to ignore this
  error use the --insecure flag.

  If you have your own certificate authority that you would like to use to
  validate the certificate chain presented by the proxy, set the
  SSL_CERT_FILE and SSL_CERT_DIR environment variables respectively and try
  again.

  If you think something malicious may be occurring, contact your Teleport
  system administrator to resolve this issue.

For reference, here's the values I deployed that with:

chartMode: gcp
clusterName: teleport.mattusaur.com
authenticationType: github
teleportVersionOverride: "9.0.0"
gcp:
  projectId: matt-melgard-personal
  backendTable: teleport-helm-backend
  auditLogTable: teleport-helm-events
  sessionRecordingBucket: mattmelgard-teleport-helm-sessions
highAvailability:
  replicaCount: 2
tls:
  existingSecretName: cloudflare-origin-cert
annotations:
  service:
    external-dns.alpha.kubernetes.io/hostname: "teleport.mattusaur.com,*.teleport.mattusaur.com"

And here's the origin certificate I created in my personal Cloudflare tenant.

This may just be a quirk of Cloudflare's Origin Certificates, but I'm still kind of digging into it to be sure. That or maybe I do need to set those env vars?

[webvictim]

@mattmelgard Thanks for testing. I'm unsure exactly what Cloudflare's origin certificates look like, but my guess is that their chain of trust is Root CA -> Intermediate CA -> Origin cert and the bundle you download only contains the Intermediate CA and Origin certs. In this situation the Root CA cert would need to be added using existingSecretNameCA so that the pods know to trust it.

I'm going to do a little testing with this myself to confirm and maybe update the instructions. I have a feeling that the cert parser which backs Teleport's https_keypairs isn't capable of reading a file containing a concatenated "Origin cert/Intermediate CA cert/Root CA cert" set, which is why we need this workaround.

@zmb3 Any thoughts on this? It looks like the downloaded file only contains the origin cert and not the root CA, so disregard. The origin root CA isn't installed via ca-certificates, and I wouldn't expect this to work without a centrally trusted root.

[webvictim]

@mattmelgard Looks like it works if you download Cloudflare's origin root CA (as per these instructions: https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/#4-required-for-some-add-cloudflare-origin-ca-root-certificates - get the one named "Cloudflare Origin RSA PEM") and add it as a CA secret:

wget https://developers.cloudflare.com/ssl/static/origin_ca_rsa_root.pem
kubectl create secret generic cloudflare-origin-cert-root-ca --from-file=ca.pem=origin_ca_rsa_root.pem

Add this to your values and do a helm upgrade:

tls:
  existingSecretNameCA: cloudflare-origin-cert-root-ca

The issue with the 10.0.0-dev image not existing is expected when deploying from a PR forked from master - i should have warned you. Using teleportVersionOverride was the correct approach.

[mattmelgard]

@mattmelgard Looks like it works if you download Cloudflare's origin root CA (as per these instructions: https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/#4-required-for-some-add-cloudflare-origin-ca-root-certificates - get the one named "Cloudflare Origin RSA PEM") and add it as a CA secret:

wget https://developers.cloudflare.com/ssl/static/origin_ca_rsa_root.pem
kubectl create secret generic cloudflare-origin-cert-root-ca --from-file=ca.pem=origin_ca_rsa_root.pem

Add this to your values and do a helm upgrade:

tls:
  existingSecretNameCA: cloudflare-origin-cert-root-ca

The issue with the 10.0.0-dev image not existing is expected when deploying from a PR forked from master - i should have warned you. Using teleportVersionOverride was the correct approach.

Thanks for getting back so quickly, I appreciate the update!

Seems you have it working so that's awesome. I'll validate on my end as well later tonight just to give you the extra data point, but if all goes well then this seems like a perfect solution for us. Thanks again for digging into this!

[mattmelgard]

@webvictim were you able to get that solution to work with a Cloudflare proxied DNS record? Seems the pods stand up okay, but I'm getting a 522 error when Cloudflare tries to connect to my backend load balancer for some reason. Might just be something misconfigured on my end, but the setup I have doesn't seem any different from other GCP managed endpoints that I've exposed using the Cloudflare origin certs/proxied DNS in the past. I'll try to dig in and debug a little more tomorrow though.

[webvictim]

@mattmelgard Afraid I haven't been that far down the rabbit hole with Cloudflare - if the pod started once the root CA was provided then that pretty much satisfies my requirement. I suspect it might be some kind of configuration issue somewhere. Do let me know how you get on, and thanks for the existing testing!