tls: failed to verify certificate: x509: certificate signed by unknown authority

[yidongw]

Expected behavior:

Access the self-hosted teleport cluster domain from my browser, and expect it to work

Current behavior:

Seeing errors on the domain page

Info for debugging:

I'm trying to use the TLS cert from Cloudflare. If I don't use the existing TLS, it would work.

Below is what I did.

used teleport/teleport-cluster chart with version: 13.0.0

followed the instructions in here and here created 2 secrets files.

My values file for teleport-cluster release

clusterName: xxx.test.dev
tls:
  existingSecretName: "my-tls-secret"
  existingCASecretName: "cloudflare-origin-cert-root-ca"
proxy:
  highAvailability:
    replicaCount: 1

Both 2 secrets are there and all pods are running okay

$ kubectl get all  -n teleport-cluster
NAME                                         READY   STATUS    RESTARTS   AGE
pod/teleport-cluster-auth-5dd46ddf7f-fjgs2   1/1     Running   0          77m
pod/teleport-cluster-proxy-999ff649b-k7s4h   1/1     Running   0          77m

NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                     AGE
service/teleport-cluster            LoadBalancer   10.128.121.27    45.79.231.99   443:31347/TCP,3023:32580/TCP,3026:30128/TCP,3024:31873/TCP,3036:31768/TCP   77m
service/teleport-cluster-auth       ClusterIP      10.128.103.181   <none>         3025/TCP,3026/TCP                                                           77m
service/teleport-cluster-auth-v12   ClusterIP      None             <none>         <none>                                                                      77m
service/teleport-cluster-auth-v13   ClusterIP      None             <none>         <none>                                                                      77m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/teleport-cluster-auth    1/1     1            1           77m
deployment.apps/teleport-cluster-proxy   1/1     1            1           77m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/teleport-cluster-auth-5dd46ddf7f   1         1         1       77m
replicaset.apps/teleport-cluster-proxy-999ff649b   1         1         1       77m

After I setup the proxy DNS record in Cloudflare

My domain in Cloudflare is matching whatever value I set in the clusterName in the values file for teleport-cluster. Access my set-up domain in my browser, but the proxy pod throws errors.

Logs from proxy pod

$ kubectl logs pod/teleport-cluster-proxy-999ff649b-k7s4h -n teleport-cluster
2023-05-10T08:59:17Z INFO             Starting Teleport v13.0.0 with a config file located at "/etc/teleport/teleport.yaml" common/teleport.go:548
2023-05-10T08:59:17Z INFO [PROC:1]    Generating new host UUID: 21900c4b-97b6-433c-83d4-3f6828a15f03. pid:7.1 service/service.go:5349
2023-05-10T08:59:18Z INFO [PROC:1]    Service diag is creating new listener on 0.0.0.0:3000. pid:7.1 service/signals.go:215
2023-05-10T08:59:18Z INFO [DIAG:1]    Starting diagnostic service on 0.0.0.0:3000. pid:7.1 service/service.go:2859
2023-05-10T08:59:18Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:18Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:18Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:18Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:18Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:18Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:18Z INFO [AUTH]      Attempting registration via proxy server. auth/register.go:271
2023-05-10T08:59:18Z INFO [AUTH]      Attempting registration via proxy server. auth/register.go:271
2023-05-10T08:59:18Z ERRO [PROC:1]    Instance failed to establish connection to cluster: dial tcp 10.128.103.181:3025: connect: connection refused, Post "https://teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025/v1/webapi/host/credentials": dial tcp 10.128.103.181:3025: connect: connection refused. pid:7.1 service/connect.go:119
2023-05-10T08:59:18Z ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:7.1 service/connect.go:121
2023-05-10T08:59:18Z ERRO [PROC:1]    Proxy failed to establish connection to cluster: dial tcp 10.128.103.181:3025: connect: connection refused, Post "https://teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025/v1/webapi/host/credentials": dial tcp 10.128.103.181:3025: connect: connection refused. pid:7.1 service/connect.go:119
2023-05-10T08:59:18Z ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:7.1 service/connect.go:121
2023-05-10T08:59:21Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:21Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:21Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:21Z INFO [AUTH]      Attempting registration via proxy server. auth/register.go:271
2023-05-10T08:59:21Z ERRO [PROC:1]    Instance failed to establish connection to cluster: dial tcp 10.128.103.181:3025: connect: connection refused, Post "https://teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025/v1/webapi/host/credentials": dial tcp 10.128.103.181:3025: connect: connection refused. pid:7.1 service/connect.go:119
2023-05-10T08:59:21Z ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:7.1 service/connect.go:121
2023-05-10T08:59:23Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:23Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:23Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:23Z INFO [AUTH]      Attempting registration via proxy server. auth/register.go:271
2023-05-10T08:59:24Z ERRO [PROC:1]    Proxy failed to establish connection to cluster: dial tcp 10.128.103.181:3025: connect: connection refused, Post "https://teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025/v1/webapi/host/credentials": dial tcp 10.128.103.181:3025: connect: connection refused. pid:7.1 service/connect.go:119
2023-05-10T08:59:24Z ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:7.1 service/connect.go:121
2023-05-10T08:59:41Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:41Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:41Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:41Z INFO [AUTH]      Attempting registration via proxy server. auth/register.go:271
2023-05-10T08:59:41Z ERRO [PROC:1]    Proxy failed to establish connection to cluster: dial tcp 10.128.103.181:3025: connect: connection refused, Post "https://teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025/v1/webapi/host/credentials": dial tcp 10.128.103.181:3025: connect: connection refused. pid:7.1 service/connect.go:119
2023-05-10T08:59:41Z ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:7.1 service/connect.go:121
2023-05-10T08:59:44Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T08:59:44Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T08:59:44Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T08:59:44Z INFO [AUTH]      Successfully registered with auth server. auth/register.go:278
2023-05-10T08:59:44Z INFO [PROC:1]    Instance has obtained credentials to connect to the cluster. pid:7.1 service/connect.go:642
2023-05-10T08:59:44Z INFO [PROC:1]    The process successfully wrote the credentials and state of Instance to the disk. pid:7.1 service/connect.go:683
2023-05-10T08:59:44Z INFO [PROC:1]    Instance: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true  pid:7.1 service/connect.go:92
2023-05-10T08:59:44Z INFO [INSTANCE:] Successfully registered instance client. pid:7.1 service/service.go:2227
2023-05-10T09:00:06Z INFO [PROC:1]    Joining the cluster with a secure token. pid:7.1 service/connect.go:585
2023-05-10T09:00:06Z INFO [AUTH]      Attempting registration with auth server. auth/register.go:271
2023-05-10T09:00:06Z WARN [AUTH]      Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:490
2023-05-10T09:00:06Z INFO [AUTH]      Successfully registered with auth server. auth/register.go:278
2023-05-10T09:00:06Z INFO [PROC:1]    Proxy has obtained credentials to connect to the cluster. pid:7.1 service/connect.go:642
2023-05-10T09:00:06Z INFO [PROC:1]    The process successfully wrote the credentials and state of Proxy to the disk. pid:7.1 service/connect.go:683
2023-05-10T09:00:06Z INFO [PROC:1]    Proxy: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true  pid:7.1 service/connect.go:92
2023-05-10T09:00:06Z INFO [PROXY:1:C] Cache "proxy" first init succeeded. cache/cache.go:863
2023-05-10T09:00:06Z INFO [PROC:1]    Service proxy:ssh is creating new listener on 0.0.0.0:3023. pid:7.1 service/signals.go:215
2023-05-10T09:00:06Z INFO [PROC:1]    Service proxy:kube is creating new listener on 0.0.0.0:3026. pid:7.1 service/signals.go:215
2023-05-10T09:00:06Z INFO [PROC:1]    Service proxy:mysql is creating new listener on 0.0.0.0:3036. pid:7.1 service/signals.go:215
2023-05-10T09:00:06Z INFO [PROC:1]    Service proxy:web is creating new listener on 0.0.0.0:3080. pid:7.1 service/signals.go:215
2023-05-10T09:00:06Z INFO [PROC:1]    Service proxy:tunnel is creating new listener on 0.0.0.0:3024. pid:7.1 service/signals.go:215
2023-05-10T09:00:06Z INFO [PROXY:SER] Reverse tunnel service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3024. pid:7.1 utils/cli.go:286
2023-05-10T09:00:06Z INFO [PROXY:SER] Starting 13.0.0:api/v13.0.0-0-g6d0ce2b on 0.0.0.0:3024 using in-memory cache pid:7.1 service/service.go:3621
2023-05-10T09:00:06Z INFO [PROXY:CER] Starting periodic reloading of certificate key pairs every 12h0m0s. service/certreloader.go:79
2023-05-10T09:00:06Z INFO [PROXY:SER] Web proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3080. pid:7.1 utils/cli.go:286
2023-05-10T09:00:06Z INFO [PROXY:SER] Web proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3080. pid:7.1 service/service.go:3791
2023-05-10T09:00:06Z INFO             debug -> starting legacy heartbeat. regular/sshserver.go:877
2023-05-10T09:00:06Z INFO [PROXY:SER] TLS multiplexer is starting on 0.0.0.0:3024. pid:7.1 service/service.go:4416
2023-05-10T09:00:06Z INFO [PROXY:SER] Minimal web proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3024. pid:7.1 utils/cli.go:286
2023-05-10T09:00:06Z INFO [PROXY:SER] Minimal web proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3024. pid:7.1 service/service.go:4443
2023-05-10T09:00:06Z INFO [PROXY:SER] SSH proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on 0.0.0.0:3023. pid:7.1 utils/cli.go:286
2023-05-10T09:00:06Z INFO [PROXY:SER] SSH proxy service 13.0.0:api/v13.0.0-0-g6d0ce2b is starting on {0.0.0.0:3023 tcp } pid:7.1 service/service.go:3979
2023-05-10T09:00:06Z INFO [UPLOAD:1]  starting upload completer service pid:7.1 service/service.go:2557
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload/streaming. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload/streaming/default. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload/corrupted. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [UPLOAD:1]  Creating directory /var/lib/teleport/log/upload/corrupted/default. pid:7.1 service/service.go:2597
2023-05-10T09:00:06Z INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. pid:7.1 service/connect.go:694
2023-05-10T09:00:06Z INFO [DB:SERVIC] Starting Database Postgres proxy server on [::]:3080. pid:7.1 service/service.go:4188
2023-05-10T09:00:06Z INFO [PROXY:AGE] Starting reverse tunnel agent pool. service/service.go:4023
2023-05-10T09:00:06Z INFO [PROXY:PRO] Starting Kube proxy on 0.0.0.0:3026. service/service.go:4098
2023-05-10T09:00:06Z INFO [DB:SERVIC] Starting MySQL proxy server on 0.0.0.0:3036. pid:7.1 service/service.go:4197
2023-05-10T09:00:06Z INFO [DB:SERVIC] Starting Database TLS proxy server on 0.0.0.0:3080. pid:7.1 service/service.go:4206
2023-05-10T09:00:06Z INFO [PROC:1]    Starting proxy gRPC server on [::]:3080. pid:7.1 service/service.go:5479
2023-05-10T09:00:06Z INFO [PROC:1]    Starting proxy gRPC server on [::]:3080. pid:7.1 service/service.go:5557
2023-05-10T09:00:06Z INFO [PROXY:SER] Starting TLS ALPN SNI proxy server on [::]:3080. pid:7.1 service/service.go:4282
2023-05-10T09:14:19Z WARN [ALPN:PROX] Failed to handle client connection. error:[
ERROR REPORT:
Original Error: *tls.CertificateVerificationError tls: failed to verify certificate: x509: certificate signed by unknown authority
Stack Trace:
	github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:392 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).handleConn
	github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:326 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).Serve.func1
	runtime/asm_amd64.s:1598 runtime.goexit
User Message: tls: failed to verify certificate: x509: certificate signed by unknown authority] alpnproxy/proxy.go:337

How can I fix it? Thanks very much

[webvictim]

First, try setting proxyListenerMode: multiplex in your values.yaml and then running helm upgrade. Multiplexing behind an HTTPS LB should work fine in Teleport 13 and I think this is a good use case.

After doing this, what happens when you visit https://teleport.<cloudflare-domain> in your browser?

[yidongw]

Thank you for replying. After setting it up, it didn't do anything. accessing the same URL still gives me the same error message

[webvictim]

What is the error message you get in the browser?

[yidongw]

Web server is returning an unknown error Error code 520
The error message we are seeing on the browser is not very helpful. The root error is in the proxy logs

[webvictim]

It doesn't look like your cluster name matches your Cloudflare DNS record - your DNS record is for teleport.<something> while your cluster name is xxx.test.dev. This is probably going to result in an incorrect host header being sent to Teleport.

You mayneed to set the public address in your values to match the full FQDN on the DNS record and then redeploy:

publicAddr: ['teleport.xxx.test.dev:443']

Also, can you share the output of these commands (with the base64 payloads/values redacted):

• kubectl -n <namespace> get secret/my-tls-secret -o yaml
• kubectl -n <namespace> get secret/cloudflare-origin-cert-root-ca -o yaml

[yidongw]

Sorry I'm just giving you an example. My DNS record is for teleport.xxx.dev, and my cluster name is teleport.xxx.dev. They are the same. xxx is a redacted company name. I double-checked again. I also check the teleport-cluster-proxy cm and made sure that they are the same.

cloudflare-origin-cert-root-ca is public info, so you can see the full data.

$ kubectl -n teleport-cluster get secret/cloudflare-origin-cert-root-ca -o yaml
apiVersion: v1
data:
  ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVBRENDQXVpZ0F3SUJBZ0lJRCtyT1NkVEdmR2N3RFFZSktvWklodmNOQVFFTEJRQXdnWXN4Q3pBSkJnTlYKQkFZVEFsVlRNUmt3RndZRFZRUUtFeEJEYkc5MVpFWnNZWEpsTENCSmJtTXVNVFF3TWdZRFZRUUxFeXREYkc5MQpaRVpzWVhKbElFOXlhV2RwYmlCVFUwd2dRMlZ5ZEdsbWFXTmhkR1VnUVhWMGFHOXlhWFI1TVJZd0ZBWURWUVFICkV3MVRZVzRnUm5KaGJtTnBjMk52TVJNd0VRWURWUVFJRXdwRFlXeHBabTl5Ym1saE1CNFhEVEU1TURneU16SXgKTURnd01Gb1hEVEk1TURneE5URTNNREF3TUZvd2dZc3hDekFKQmdOVkJBWVRBbFZUTVJrd0Z3WURWUVFLRXhCRApiRzkxWkVac1lYSmxMQ0JKYm1NdU1UUXdNZ1lEVlFRTEV5dERiRzkxWkVac1lYSmxJRTl5YVdkcGJpQlRVMHdnClEyVnlkR2xtYVdOaGRHVWdRWFYwYUc5eWFYUjVNUll3RkFZRFZRUUhFdzFUWVc0Z1JuSmhibU5wYzJOdk1STXcKRVFZRFZRUUlFd3BEWVd4cFptOXlibWxoTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQwpBUUVBd0VpVlovVW9RcEhtRnNIdms1aXNCeFJlaHVrUDhERzlKaEZldjNXWnRHNzZXb1R0aHZMSkZSS0ZDSFhtClY2WjUvNjZaNFMwOW1nc1V1Rnd2SnpNbkU2RWo2eUlzWU5DYjlyOVFPUmE4QmRocmtObjZrZFRseTNtZG55a2IKT29tbndiVWZMbEV4VmdOZGxQMFhvUm9lTXdiUTQ1OThmb2lIYmxPMkIvTEt1TmZKekFNZlM3b1plMzRiK3ZMQgp5clAvMWJnQ1NMZGMxQXhRYzFBQzBFc1FRaGdjeVRKTmduRzR2YTFjN29nUGx3S3loYkR5WjRlNTlONWxiWVBKClNtWEkvY0FlM2pYajFGQkxKWmt3bm9ES2UwdjEzeGVGK25GMzJzbVNIMHFCN2FKWDJ0Qk1XNFRXdEZQbXpzNUkKbHdyRlN5U1dBZHdZZGd4dzE4MHlLVTBkdndJREFRQUJvMll3WkRBT0JnTlZIUThCQWY4RUJBTUNBUVl3RWdZRApWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWRCZ05WSFE0RUZnUVVKT2hUVjExOE5FQ0hxZXVVMjdyaEZuajhLYVF3Ckh3WURWUjBqQkJnd0ZvQVVKT2hUVjExOE5FQ0hxZXVVMjdyaEZuajhLYVF3RFFZSktvWklodmNOQVFFTEJRQUQKZ2dFQkFId09mOVVyMWwwQXI1dkZFNlBOclpXckRmUUlNeUVmZGdTS29mQ2RUY2ticVhOVGlYZGdiSHMrVFdvUQp3QUIwcGZKREFISkRYT1RDV1J5VGVYT3NlZU9pNUJ0ajVDbkV1dzNQMG9YcWRxZXZNMS8rdVdwMENNMzV6Z1o4ClZENGFJVHhpdHkwZGp6RTZRbngzU3l6eitaa29CZ1RuTnVtN2Q5QTY2L1Y2MzZ4NHZUZXFiWkZCcjllckp6Z3oKaGh1cmpjb2FjdlJOaG5qdERSTTBkUGVpQ0o1MENQM3dFWXV2VXpESFVhb3dPc25MQ2pRSWtXYlI3Tmk2S0VJawpNT3oyVTBPQlNpZjNGVGtoQ2daV1FLT09MbzFQNDJqSEMzc3NVWkF0Vk5YckNrM2Z3OS9FMTVrOE5Qa0Jhelo2CjBpeWtMaEgxdHJ5d3JLUk1WdzY3RjQ0SUU4WT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
kind: Secret
metadata:
  creationTimestamp: "2023-05-10T08:29:32Z"
  name: cloudflare-origin-cert-root-ca
  namespace: teleport-cluster
  resourceVersion: "296315"
  uid: 191ec934-138e-4c8e-baeb-6375ac99f648
type: Opaque

$ kubectl -n teleport-cluster get secret/my-tls-secret -o yaml
apiVersion: v1
data:
  tls.crt: <redacted>
  tls.key: <redacted>
kind: Secret
metadata:
  creationTimestamp: "2023-05-10T08:49:28Z"
  name: my-tls-secret
  namespace: teleport-cluster
  resourceVersion: "300072"
  uid: 9246196c-37a9-4976-af28-4ae198edba98
type: kubernetes.io/tls

[webvictim]

OK, cool, that should be fine. Can you also share the output of kubectl -n teleport-cluster get configmap/<release-name>-proxy -o yaml and kubectl -n teleport-cluster describe deployment/<release-name>-proxy too please? I'm trying to make sure that the TLS cert/key and the origin CA are being provided to the pod correctly.

[yidongw]

$ kubectl get cm -n teleport-cluster teleport-cluster-proxy -o yaml
apiVersion: v1
data:
  teleport.yaml: |
    auth_service:
      enabled: false
    proxy_service:
      enabled: true
      https_keypairs:
      - cert_file: /etc/teleport-tls/tls.crt
        key_file: /etc/teleport-tls/tls.key
      https_keypairs_reload_interval: 12h
      public_addr: teleport.xxx.dev:443
    ssh_service:
      enabled: false
    teleport:
      auth_server: teleport-cluster-auth.teleport-cluster.svc.cluster.local:3025
      join_params:
        method: kubernetes
        token_name: teleport-cluster-proxy
      log:
        format:
          extra_fields:
          - timestamp
          - level
          - component
          - caller
          output: text
        output: stderr
        severity: INFO
    version: v3
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: teleport-cluster
    meta.helm.sh/release-namespace: teleport-cluster
  creationTimestamp: "2023-05-12T06:44:07Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: teleport-cluster-proxy
  namespace: teleport-cluster
  resourceVersion: "777425"
  uid: 28f19e06-3e3b-40a0-80c6-2aa2d993ce8b

$ kubectl -n teleport-cluster describe deployment/teleport-cluster-proxy
Name:                   teleport-cluster-proxy
Namespace:              teleport-cluster
CreationTimestamp:      Fri, 12 May 2023 14:44:08 +0800
Labels:                 app.kubernetes.io/component=proxy
                        app.kubernetes.io/instance=teleport-cluster
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=teleport-cluster
                        app.kubernetes.io/version=13.0.0
                        helm.sh/chart=teleport-cluster-13.0.0
                        teleport.dev/majorVersion=13
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: teleport-cluster
                        meta.helm.sh/release-namespace: teleport-cluster
Selector:               app.kubernetes.io/component=proxy,app.kubernetes.io/instance=teleport-cluster,app.kubernetes.io/name=teleport-cluster
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        15
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=proxy
                    app.kubernetes.io/instance=teleport-cluster
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=teleport-cluster
                    app.kubernetes.io/version=13.0.0
                    helm.sh/chart=teleport-cluster-13.0.0
                    teleport.dev/majorVersion=13
  Annotations:      checksum/config: 452f1cd60077d4c56fc1109119e1a044692145592aa0fcccd6aa484825cf85c9
  Service Account:  teleport-cluster-proxy
  Init Containers:
   wait-auth-update:
    Image:      public.ecr.aws/gravitational/teleport-distroless:13.0.0
    Port:       <none>
    Host Port:  <none>
    Command:
      teleport
      wait
      no-resolve
      teleport-cluster-auth-v12.teleport-cluster.svc.cluster.local
    Environment:  <none>
    Mounts:       <none>
  Containers:
   teleport:
    Image:       public.ecr.aws/gravitational/teleport-distroless:13.0.0
    Ports:       3080/TCP, 3000/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      --diag-addr=0.0.0.0:3000
    Liveness:   http-get http://:diag/healthz delay=5s timeout=1s period=5s #success=1 #failure=6
    Readiness:  http-get http://:diag/readyz delay=5s timeout=1s period=5s #success=1 #failure=12
    Environment:
      SSL_CERT_FILE:  /etc/teleport-tls-ca/ca.pem
    Mounts:
      /etc/teleport from config (ro)
      /etc/teleport-tls from teleport-tls (ro)
      /etc/teleport-tls-ca from teleport-tls-ca (ro)
      /var/lib/teleport from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from proxy-serviceaccount-token (ro)
  Volumes:
   proxy-serviceaccount-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3600
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
   teleport-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-tls-secret
    Optional:    false
   teleport-tls-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudflare-origin-cert-root-ca
    Optional:    false
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      teleport-cluster-proxy
    Optional:  false
   data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   teleport-cluster-proxy-86b69d5768 (1/1 replicas created)
Events:          <none>

[webvictim]

Yep, looks good to me. I guess the issue is definitely between Cloudflare and the Teleport load balancer.

What happens if you run curl -kLv --resolve teleport.xxx.dev:443:45.79.231.99 https://teleport.xxx.dev?

Alternatively, try disabling the Cloudflare forwarding for the DNS record and have it connect directly - what happens?

We need to make sure that the load balancer itself is actually responding correctly with a Teleport page before adding the Cloudflare element in.

[yidongw]

$ curl -kLv --resolve teleport.xxx.dev:443:45.79.231.99 https://teleport.xxx.dev
* Added teleport.xxx.dev:443:45.79.231.99 to DNS cache
* Hostname teleport.xxx.dev was found in DNS cache
*   Trying 45.79.231.99:443...
* connect to 45.79.231.99 port 443 failed: Operation timed out
* Failed to connect to teleport.xxx.dev port 443 after 75000 ms: Operation timed out
* Closing connection 0
curl: (28) Failed to connect to teleport.xxx.dev port 443 after 75000 ms: Operation timed out

If I ping teleport.xxx.dev, it will not reply with 45.79.231.99. It is because in order to use the Cloudflare TLS, I need to also use Cloudflare proxy DNS. All the traffic will go to the teleport.xxx.dev proxy server first and then be proxied to 45.79.231.99

If I switch off proxy DNS, all the traffic will go directly to 45.79.231.99 and I will have the following error

NET::ERR_CERT_AUTHORITY_INVALID
Subject: CloudFlare Origin Certificate

Issuer: CloudFlare, Inc.

Expires on: May 6, 2038

Current date: May 14, 2023

PEM encoded chain:
-----BEGIN CERTIFICATE-----
<redacted>
-----END CERTIFICATE-----

You can also give it a try yourself.

[webvictim]

I can't hit that address myself, it doesn't respond:

gus@hades:~$ curl -v https://45.79.231.99
*   Trying 45.79.231.99:443...

Is there a firewall or something in place to prevent direct connections?

The response you see without the Cloudflare proxying looks like exactly what I'd expect to see, so it seems like maybe there's a connection issue between Cloudflare and the Kubernetes LB?

[yidongw]

I can't hit that address myself, it doesn't respond:

I was suggesting maybe you could get a Cloudflare account and try to replicate what I'm doing

Is there a firewall or something in place to prevent direct connections?

because I used the Cloudflare mTLS, so 45.79.231.99 is only allowing direct access from Cloudflare.

The response you see without the Cloudflare proxying looks like exactly what I'd expect to see, so it seems like maybe there's a connection issue between Cloudflare and the Kubernetes LB?

I'm not sure what you mean by connection issue. Cloudflare can access Kubernetes LB and the traffic is hitting the teleport proxy pod, so the pod will generate error messages.

[yidongw]

@webvictim Please let me know when you have some updates. Thank you so much

[webvictim]

After investigation, I have found that Teleport does not work correctly when deployed behind "proxied" Cloudflare DNS records. This is due to the fact that Cloudflare does not support Teleport's custom ALPN upgrades through its proxy, only standard websocket upgrades. An issue to work around this behaviour is tracked here on the Teleport side: #30493

If you switch to "DNS only" records on Cloudflare, open up traffic to the internet and still experience connectivity issues, please reopen this issue.