kubernetes-crd with tls doesn't work

[Roydon]

Do you want to request a feature or report a bug?

Bug

What did you do?

Want to provide SSL certificate through secrets for specific hosts.

What did you expect to see?

Traefik to use provided SSL certificate for handshake.

What did you see instead?

Traefik is using TRAEFIK DEFAULT CERT not the provided one.

Output of traefik version: (What version of Traefik are you using?)

Version:      2.0.5
Codename:     montdor
Go version:   go1.13.4
Built:        2019-11-14T18:11:01Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Traefik is running in Kubernetes (EKS) as a Daemon Sets.

    ## static configuration
    [global]
      checkNewVersion = true

    [entryPoints]
      [entryPoints.web]
        address = ":80"
      [entryPoints.websecure]
        address = ":443"
      [entryPoints.internal]
        address = ":4080"
      [entryPoints.internalsecure]
        address = ":4083"
      [entryPoints.tcp5050]
        address = ":5050"

    [providers]
      [providers.kubernetesCRD]
      [providers.file]
        directory = "/etc/traefik/providers/"
        watch = true

    [log]
      level = "INFO"

    [accessLog]

    [api]
      insecure = true
      dashboard = true
      debug = true

    [metrics]
      [metrics.prometheus]
        buckets = [0.1,0.3,1.2,5.0]
        addEntryPointsLabels = true
        addServicesLabels = true
        entryPoint = "web"

    [ping]
      entryPoint = "web"

    [tls.certificates]
      certFile = "/certs/star_domain_com.crt"
      keyFile = "/certs/star_domain_com.key"

    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile = "/certs/star_domain_com.crt"
          keyFile  = "/certs/star_domain_com.key"

    [certificatesResolvers]
      [certificatesResolvers.default]
        [certificatesResolvers.default.acme]
          email = "[email protected]"
          caServer = "https://acme-v02.api.letsencrypt.org/directory"
          storage = "/etc/traefik/storage/acme.json"
          [certificatesResolvers.default.acme.dnsChallenge]
            provider = "route53"
            delayBeforeCheck = 0
            resolvers = ["1.1.1.1:53", "8.8.8.8:53"]

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Certificate provided as Secret

apiVersion: v1
kind: Secret
metadata:
  name: star-domain-net
  namespace: kube-system
data:
  star_domain_net.crt: "xxxxxx"
  star_domain_net.key: "xxxxxx"

IngressRoute for host util.domain.net as per doc

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: firmware
  namespace: dev
spec:
  entryPoints:
    - internal
    - internalsecure
  routes:
    - match: Host(`util.domain.net`)
      kind: Rule
      services:
        - name: firmware
          port: 80
  tls:
    secretName: star-domain-net
    namespace: kube-system

Then I tried openssl s_client -showcerts -servername util.domain.net -connect util.domain.net:443 but getting following message with TRAEFIK DEFAULT CERT

CONNECTED(00000003)
depth=0 CN = TRAEFIK DEFAULT CERT
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = TRAEFIK DEFAULT CERT
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=TRAEFIK DEFAULT CERT
   i:/CN=TRAEFIK DEFAULT CERT
-----BEGIN CERTIFICATE-----
MIIDXjCCAkagAwIBAgIRALNbpwoNofbzCLeFOBveiOMwDQYJKoZIhvcNAQELBQAw
.
.
.
e6im/t3diQvJLWPFObUy+T3d5HMWfQxrKS+UFEUaLJKrFFBl3VR7iXEj31qDItcf
KtU=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=TRAEFIK DEFAULT CERT
issuer=/CN=TRAEFIK DEFAULT CERT
---
No client certificate CA names sent
---
SSL handshake has read 1472 bytes and written 454 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 8FDB23A6B5...5D4CBA541715
    Session-ID-ctx:
    Master-Key: 8EABA2D....4ACDC8963
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - eb d9 66 dd c3 fd bc 1b-38 eb cd b4 72 00 6d 3e   ..f.....8...r.m>
    0010 - 22 fa 1e 01 a6 6b 0f 19-c1 86 8c 40 f5 33 61 40   "[email protected]@
    0020 - 8d 98 43 03 c8 23 25 eb-93 9e e9 9c ef 42 19 6b   ..C..#%......B.k
    0030 - 64 fd ed 44 f2 36 8b c4-70 21 91 4d 5b 32 da 5a   d..D.6..p!.M[2.Z
    0040 - 69 8b f0 20 db 72 68 2b-a4 f8 c8 96 af 40 49 73   i.. .rh+.....@Is
    0050 - 97 f4 8f 37 01 5d 80 20-a2 9e e9 ce dc 23 5b da   ...7.]. .....#[.
    0060 - 9b b6 82 9f 99 95 62 f2-f7 20 d0 bf bf 04 bf 35   ......b.. .....5
    0070 - 3f a4 f1 05 77 4a 1d fb-                          ?...wJ..

    Start Time: 1574808229
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

I was expecting Traefik to return the certificate I have configured via secret (star-domain-net)

[dduportal]

Hi @Roydon , the keyword namespace: is not existing directly under the tls: key: https://docs.traefik.io/v2.0/reference/dynamic-configuration/kubernetes-crd/
The only namespace: reference is for the tls.options object.

Please not that the standard "Ingress" objects have the same behavior: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#ingresstls-v1beta1-networking-k8s-io .

It means that the field is not part of the API objects: this is why Traefik cannot get the secret: the Kubernetes security models forbids ingress to get secrets from other namespaces (you have the same limitation with other ingresses as Nginx: kubernetes/ingress-nginx#2170 ).

As Traefik cannot find any certificates because the secret does not exist, then it serves the default cert to ensure the connexion is still in HTTPS.

From there you have the following solutions:

• If you can afford storing the secret with the certificate in the same namespace as the ingress (either by moving the ingress, or the secret), then it should solve the issue by removing the tls.namespace directive.
• If you can provide the secret to Traefik (it looks like, but I cannot be sure since you did not provide any DaemonSet manifest neither any debug logs from Traefik that could help us investigate). But it the certificate is correctly loaded in Traefik, then you only have to update your ingress with tls: {}: Traefik will pick the hostname from the IngressRoute, and then will match the certificate configured under its file provider.

[Roydon]

@dduportal Does traefik checks if Domain in Host(util.domain.net) matches with certificate common name domain (*.domain.net) ?

[dduportal]

@Roydon yes it does. If it is not what you see, then you have a misconfiguration and providing more elements (I'm thinking about the debug logs of Traefik) could help to find the misconfiguration element :)

May I ask you to switch to the community forum at https://community.containo.us/: you would benefit from the community expertise for a faster help, and it would help us to manage the issue triage here, as we use the issues for bugs and feature requests.

Of course, if the topic you'll open in the community forum raises a reproductible bug, then we'll re-open the issue here with more context.

Thanks a lot!