Referring to TLS secret from other namespace (i.e. not the namespace in which ingress is created)

[amit-kumar-4]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): "Secret namespace"

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature

NGINX Ingress controller version: "0.10.2"

Kubernetes version (use kubectl version): v1.8.7

Environment:

• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): Ubuntu 16.04.3
• Kernel (e.g. uname -a): 4.11.0-1016-azure
• Install tools: Helm Chart v0.9.5 from https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/
• Others:

What happened:
When trying to use a TLS certificate using <namespace>/<secretName> pattern in tls section of ingress definition, Nginx controller still tries to get the details from the namespace where ingress was created.

What you expected to happen:
When TLS secret referred when creating an ingress is of pattern <namespace>/<secretName>, ingress controller shouldn't check only in ingress' namespace.

How to reproduce it (as minimally and precisely as possible):

1. Create 2 namespaces, say secret-store and ingress-store.
2. Create a secret containing a TLS certificate and key in secret-store namespace, say my-tls.
3. Create an ingress in ingress-store namespace with TLS enabled and in the .spec.tls.hosts[].secretName field put secret-store/my-tls to refer to the secret in secret-store namespace.
4. Check logs of ingress controller, line similar to below will be printed, indicating secret was never searched in secret-store namespace:

W0305 11:39:26.826578       6 backend_ssl.go:49] error obtaining PEM from secret ingress-store/secret-store/my-tls: error retrieving secret ingress-store/secret-store/my-tls: secret ingress-store/secret-store/my-tls was not found

Anything else we need to know:
Initial glance on the code suggests that in below snippet, in the if block, we check if the secret has a / in it, and try to extract the secret from the namespace provided in ingress .spec.tls definition, it could work.

ingress-nginx/internal/ingress/controller/controller.go

Lines 1000 to 1005 in 164bb7b

	key := fmt.Sprintf("%v/%v", ing.Namespace, tlsSecretName)
	cert, err := n.store.GetLocalSecret(key)
	if err != nil {
	glog.Warningf("ssl certificate \"%v\" does not exist in local store", key)
	continue
	}

[mikebryant]

The standard security model for Kubernetes is that secrets are only accessible from within the current Namespace.

As per the API definition, the secretName field is purely a name reference, and cannot cross namespaces.

[amit-kumar-4]

True enough, but I think there is a catch-22 when trying to use wildcard certificate in cluster and keeping the certificate private key secure.
If I don't use the way mentioned above, I'll be exposing the private key for the wildcard certificate for anyone who has access to read secrets in the namespace, which will be many usually.

The standard security model for Kubernetes is that secrets are only accessible from within the current Namespace.

Since it will be the Ingress controller reading the secret, rather than something in the ingress namespace, I think your statement will still remain true.

[aledbf]

Closing. This should be allowed in the Ingress spec first. We are are not going to break this rule.

[sheerun]

@aledbf Is there a followup issue for changing the spec?

[aledbf]

@sheerun not yet

[mattalberts]

@aledbf @sheerun @aku105
During my evaluation of 0.14.0 and 0.15.0, I noticed that the ingress spec (as consumed by the nginx controller) would allow a mini-version of the issue described above without any patching (supported now).

• configure the controller with --default-ssl-certificate set to your wildcard cert
• define all ingress spec with hosts but do not specify the secretName

The ingress will be registered :80,:443 and use the default (wildcard certificate).

[Nowaker]

Thanks @mattalberts. While this is far from ideal (you can only have one default certificate), it's better than nothing.

That said, the spec should be updated to address the issue of using secrets from different namespaces. Wildcard certificates should be stored in the ingress controller's namespace, and Ingress resources from different namespaces should be allowed to reference them. What is the process of bringing up the spec discussion? Thanks! @sheerun @aledbf

[caleblloyd]

I look forward to an official fix coming in terms of a spec change that allows for referring to Secrets across namespace boundaries.

In the meantime, I am using a deployment that runs 2 kubectl containers - one to watch for new namespaces and copy the TLS Secret, and one to watch the TLS Secret for changes and apply to all namespaces. The code is here: ingress-cert-reflector.yml and I also wrote a corresponding blog post with detailed instructions.

[FossPrime]

UPDATE: in nginxinc/nginx-ingress default-ssl-certificate is default-server-tls-secret

---

@mattalberts I've tried to drop default-ssl-certificate in my controller, but I can't find where it goes

Dropping it in the spec gives ValidationError(DaemonSet.spec): unknown field "default-ssl-certificate" in io.k8s.api.extensions.v1beta1.DaemonSetSpec

my 1.2.0 controller spec

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  generation: 8
  labels:
    app: nginx-ingress
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
    spec:
      containers:
      - args:
        - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
        - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
        - -v=3
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: nginx/nginx-ingress:1.2.0
        imagePullPolicy: IfNotPresent
        name: nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress
      serviceAccountName: nginx-ingress
      terminationGracePeriodSeconds: 30
  templateGeneration: 8
  updateStrategy:
    type: OnDelete

[aledbf]

Dropping it in the spec gives ValidationError(DaemonSet.spec): unknown field "default-ssl-certificate" in io.k8s.api.extensions.v1beta1.DaemonSetSpec

You are using a different ingress controller https://github.com/nginxinc/kubernetes-ingress where that flag is not supported

[abelal83]

@aledbf @sheerun @aku105
During my evaluation of 0.14.0 and 0.15.0, I noticed that the ingress spec (as consumed by the nginx controller) would allow a mini-version of the issue described above without any patching (supported now).

• configure the controller with --default-ssl-certificate set to your wildcard cert
• define all ingress spec with hosts but do not specify the secretName

The ingress will be registered :80,:443 and use the default (wildcard certificate).

@mattalberts could you show me an example ingress you have that this works with? I've setup as you suggested but when trying to access on https I get 404 but http works. Checking the nginx.conf file on nginx it only registers port 80.

[kfox1111]

Does your ingress that's 404ing for https have a tls section with the hostname in it but without secretName? the tls section needs to be there so it knows to bind the service to https too.

[abelal83]

That did the trick! Thank you.

[mohamedfarouk]

i tried the below work around and it worked for me

• Create your secret in the desired namespace

• Create dummy ingress in the same namespace, reference the created secret normally

it seems nginx load the ingress and associated secrets, and if there any other ingress in different namespace referenced the same shared secret by name it will be already loaded by the dummy ingress so nginx will use it.

[Nowaker]

@mohamedfarouk Wow. Thanks for sharing this.

[nanorobocop]

@mohamedfarouk I'm wondering how do you reference same shared secret by name from different namespace? Could you please share your configs?

[nanorobocop]

@mohamedfarouk ok, I've figured it out and it also worked for me with following config:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dummy-load-certificates
  namespace: ingress-nginx
spec:
  tls:
  - hosts:
    - “*.example.com"
    secretName: cert-example.com
  rules:
  - host: “*.example.com"

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-app
  namespace: example-app
spec:
  tls:
  - hosts:
    - app.example.com
  rules:
    - host: app.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: example-app
            servicePort: 80

[janosi]

@aledbf would you consider an annotation like nginx.ingress.kubernetes.io/tls-secret-namespace acceptable to configure the namespace of the TLS secret in an Ingress?

[aledbf]

@janosi no, sorry. This is one of the rules defined in the ingress spec that we are not going to bend/break.

[zeeZ]

Note that the above workaround only works if the ingress containing the actual secret is the oldest one including that host. It will fall back to the default certificate and ignore all others. See also:

• Regression: Default certificate used if oldest ingress does not contain TLS secret #4926
• Ignore default certificate during server initialization TLS loop #2279

[janosi]

@aledbf I have hard times with finding the restriction in the Ingress specification https://kubernetes.io/docs/concepts/services-networking/ingress/
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#ingresstls-v1beta1-extensions
Maybe I look at the wrong specification? Thank you!

[aledbf]

@janosi Ingress predates the KEP process. For that reason, the are no formal documents besides the code. You can see the definitions and comments in types.go#L102. From the code, you can get two clear rules, service and secret are located in the same namespace. There is no way to reference a different one.
Also, Ingress exists before RBAC, something that introduces another dimension that needs to be considered, where there is no possibility to restrict access to one object.

These issues are being tackled in Ingress V2 https://github.com/kubernetes-sigs/service-apis where there is a clear delegation model and clear Roles and personas

[glothriel]

Anyone else noticed that using a hack proposed by @mohamedfarouk and @nanorobocop results in broken prometheus exporter metrics?

[countablecloud]

Anyone else noticed that using a hack proposed by @mohamedfarouk and @nanorobocop results in broken prometheus exporter metrics?

@glothriel Do you mean broken metrics on (1.) the ingress controller and/or cert manager pods or (2.) on all the deployments linked to that ingress (or 3. both...)? I'm contemplating using their hack as a method for getting around LetsEncrypt limits and I can theoretically tolerate 1 but, 2 or 3 are kind of deal breakers if it means no metrics from any of the workloads in the cluster that use this method...

[glothriel]

@countablecloud only some ingress controller metrics, afair it was connected to using wildcard hostname, not to the hack itself. We switched to haproxy due to that as ingress metrics were a must for us.

[lapwingcloud]

Hello there I'm just wondering whether it's possible to support configure certificates in the same namespace where ingress-nginx is deployed. So that ingress resources in other namespaces don't need to have the secret defined in their namespace?

The scenario is like, say, we have a lot of microservices, and a lot of tls certificates for each one. But we don't want them to be accessible to application engineers who usually have access to the microservice namespace. Because they don't need to access the TLS certificate key.

So we just want to deploy all the certificates in the ingress-nginx namespace, which also makes sense. Because it
s common scenario to deploy all certificates along with the load balancer instead of in the applications (e.g. in cloud load balancers or nginx as load balancer in virtual servers before all the microservices).

[crinjes]

You can create a dummy ingress that specifies a secret in the ingress-nginx namespace, and then elsewhere use ingresses for those domains without specifiying the secret reference. The only caveats are that this is allegedly not supported by the spec, and the ingress referencing the secret needs to be the oldest (#4926).

[lapwingcloud]

You can create a dummy ingress that specifies a secret in the ingress-nginx namespace, and then elsewhere use ingresses for those domains without specifiying the secret reference. The only caveats are that this is allegedly not supported by the spec, and the ingress referencing the secret needs to be the oldest (#4926).

I'd rather do it properly than make this hack. That's why I asked despite I saw this workaround in previous comments.

My point is, ingress nginx is built on top of nginx. Supporting to configure it in a more nginx way doesn't sound like a bad idea to me.

I might be wrong but this doesn't sound very hard to implement. Just need to define which cert maps to which host and merges it with the ingress definition.

Or this still falls into the category of "changing the ingress spec"?

[winkee01]

Any updates on this issue? As i am using cert-manager to generate certificates (in namespace cert-manager, not default, for security reasons) for my cluster, this problem is causing me a lot of headaches.

[CyanoKobalamyne]

The Contour ingress controller has explicit support for referencing certificates from other namespaces, specifically to allow a cluster administrator to create wildcard certificates to be used by other namespaces. They achieve this via a custom resource called TLSCertificateDelegation, see https://projectcontour.io/docs/v1.20.1/config/tls-delegation/. This is a rather common scenario, as evidenced by the comments above and is supported by other ingress controllers. @aledbf would you mind clarifying why nginx is not able to support something similar?

[coopbri]

@CyanoKobalamyne agreed, I have seen support for this common use case in several ingress controller implementations. I do understand the security and K8s ingress spec concerns, but the resistance is a bit strange to me considering how common this use case is. It is unfortunate that the recommended approach is to install a third-party secret syncing solution (although I respect it, in a way, in the name of separation of concerns).

Although not a direct solution, it is worth mentioning that the budding Kubernetes Gateway API has native support for cross-namespace TLS cert references. Something to keep an eye on! Lots of projects are building out implementations based on this spec (see here).

[joac]

Any news about this? Is end of 2023 and this is still a problem.

[corey-aloia]

using --default-ssl-certificate, via helm controller.extraArgsdefault-ssl-certificate works and is great for my use case, but as others have pointed out, there may be more than 1 wild card cert.

@aledbf you mention that it is a ingress defined rule that would not be broken, but when I use the ingress annotation nginx.ingress.kubernetes.io/auth-tls-secret, then this secret only needs to be in the same namespace as the ingress-nginx controller. But when using a spec.tls.secretName, it needs to be in every NS (or set as the default). So me it seems inconsistent, and like a solution could be provided where the wildcard(s) certs only need to be in the nginx controller namespaces. Maybe even also via an annotation like nginx.ingress.kubernetes.io/shared-cert-secret? Does that makes sense?

[Sumis34]

Any news on this?

[Zappelphilipp]

@Sumis34

The problem mentioned should be solved by Mergeable Ingress Resources like mentioned in this blog: https://tech.aabouzaid.com/2022/08/2-ways-to-route-ingress-traffic-across-namespaces.html

However, I'm not sure if the kubernetes-nginx controller supports this. Maybe somebody knows that.

[longwuyuan]

this controller has a flag to set the global default-cert but user going across namespace is not supported and will not be supported in future also because of security reasons.