Client Certificate Auth does not work in 0.12.0

[smelchior]

NGINX Ingress controller version: 0.12.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:06Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian Stretch
• Kernel (e.g. uname -a): 4.9.0-5-amd64
• Install tools: kops

What happened:
i have some ingresses annotated to use client certificate authentication:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-secret: p/cca
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  name: accts-cca
  namespace: p
spec:
  rules:
  - host: dummy.host
    http:
      paths:
      - backend:
          serviceName: accts
          servicePort: 80
        path: /
  tls:
  - hosts:
    - dummy.host
    secretName: tls-dummy

these work in the controller in version 0.11.0 but not in 0.12.0
What you expected to happen:
client auth to be configured for the host

How to reproduce it (as minimally and precisely as possible):
Configure an ingress with client certificate auth and check if it is enabled

Anything else we need to know:
Downgrading to 0.11.0 solved my problem, i was trying to track down the changes from 0.11.0 to 0.12.0 but did not figure out why this is not working anymore :-|

[oilbeater]

can you paste the diff of nginx.conf that generated by two version of ingress-nginx?

[ant1441]

I am also seeing this issue after upgrading from 0.11.0 to 0.12.0
A downgrade also solved the problem.

My environment is slightly different:
Kubernetes version (use kubectl version): v1.8.8-gke.0

Environment:
Cloud provider or hardware configuration: GKE

And my Ingress resource has the following annotations:

    ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    ingress.kubernetes.io/auth-tls-secret: namespace/name-ca-cert
    ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-SSL-certificate        $ssl_client_cert;
      proxy_set_header ssl-client-serial        $ssl_client_serial;
      proxy_set_header HTTP_SSL_CLIENT_M_SERIAL $ssl_client_serial;
    ingress.kubernetes.io/rewrite-target: /namespace-name
    kubernetes.io/ingress.class: nginx-public

The diff between the 0.11.0 and 0.12.0 nginx.conf looks like (just showing the server block relevant to this Ingress):

3750c3861
<         more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
---
>         location ~* / {
3752,3755c3863,3865
<         # PEM sha: 2ad310c2bd1e1be30dd283e48c073804a2c0bf72
<         ssl_client_certificate                  /ingress-controller/ssl/ca-namespace-name-ca-cert.pem;
<         ssl_verify_client                       on;
<         ssl_verify_depth                        1;
---
>             if ($scheme = https) {
>             more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
>             }
3757d3866
<         location ~* / {
3779,3782c3888,3890
<             proxy_set_header ssl-client-cert        $ssl_client_escaped_cert;
< 
<             proxy_set_header ssl-client-verify      $ssl_client_verify;
<             proxy_set_header ssl-client-dn          $ssl_client_s_dn;
---
>             proxy_set_header ssl-client-cert        "";
>             proxy_set_header ssl-client-verify      "";
>             proxy_set_header ssl-client-dn          "";

To me, that indicates that at template rendering time $server.CertificateAuth.CAFileName is empty for this Ingress.

[ant1441]

This commit removed the following line from internal/ingress/controller/controller.go:

server.CertificateAuth = anns.CertificateAuth

Made in PR #2146 by @aledbf

Due to this, the CertificateAuth field is never set.

I don't fully follow the context of that change to understand how it may be resolved.

[aledbf]

@ant1441 thank you for finding the issue. That removal was a mistake.

[aledbf]

@ant1441 please use quay.io/aledbf/nginx-ingress-controller:0.348

[ant1441]

Looks good, thanks 👍