auth-tls-pass-certificate-to-upstream does not work for GRPC

[olvesh]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): no

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): grpc cert

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: 0.13.0
Kubernetes version (use kubectl version): 1.8.8

What happened:
I have added these annotations to my ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx-grpc"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-secret: "default/grpcbin-test"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
[...]

What you expected to happen:
Using this setting I would expect the client cert to be available in the ssl-client-cert header in the backend.

How to reproduce it (as minimally and precisely as possible):
Apply grpcbin and steps from this gist (note that you need to change hosts in the tls section of the ingress): https://gist.github.com/olvesh/033445fbbe273219be2559f23dc8c65c

Anything else we need to know:
We hacked the nginx template to include the following to make it work for us, but would like a discussion on how to properly to this (grpc_set_header are the new ones):

# Pass the extracted client certificate to the backend
{{ if not (empty $server.CertificateAuth.CAFileName) }}
{{ if $server.CertificateAuth.PassCertToUpstream }}
proxy_set_header ssl-client-cert        $ssl_client_escaped_cert;
grpc_set_header  ssl-client-cert        $ssl_client_escaped_cert;
{{ end }}
proxy_set_header ssl-client-verify      $ssl_client_verify;
proxy_set_header ssl-client-subject-dn  $ssl_client_s_dn;
proxy_set_header ssl-client-issuer-dn   $ssl_client_i_dn;
grpc_set_header  ssl-client-verify      $ssl_client_verify;
grpc_set_header  ssl-client-subject-dn  $ssl_client_s_dn;
grpc_set_header  ssl-client-issuer-dn   $ssl_client_i_dn;
{{ end }}

Using this template we got the following output:

{
  "Metadata": {
    "grpc-accept-encoding": {
      "values": ["gzip"]
    },
    ":authority": {
      "values": ["default-grpcbin-80"]
    },
    "ssl-client-cert": {
      "values": ["-----BEGIN%20CERTIFICATE-----%0AMIIB%[ snip ]%0A-----END%20CERTIFICATE-----%0A"]
    },
    "ssl-client-verify": {
      "values": ["SUCCESS"]
    },
    "ssl-client-subject-dn": {
      "values": ["O=Internet Widgits Pty Ltd,ST=Some-State,C=AU"]
    },
    "ssl-client-issuer-dn": {
      "values": ["O=Internet Widgits Pty Ltd,ST=Some-State,C=AU"]
    },
    "user-agent": {
      "values": ["grpc-java-netty"]
    }
  }
}

[aledbf]

@olvesh thank you for the report. Right now we don't set headers for gRPC. I think we just need to add a helper to check if the proxy_pass directive will be grpc_pass or not and change proxy_set_header to grpc_set_header

[olvesh]

Sure, our fix was a hack to see if it worked, not intended as a proposal for a solution.

Is there a variable available in the template to signal that an grpc_pass is active for the server{..} directive? I see that the grpc_pass directive is built in the go code, but it wasn't immediately apparent if that fact was available in the template via the TemplateConfig struct.

[aledbf]

Is there a variable available in the template to signal that an grpc_pass is active for the server{..} directive?

No

[olvesh]

I don't immediately see how much work this is, not too familiar with the ingress-nginx codebase, but I can probably take a stab at this later in the week / next week.

Would be very glad for any pointers on how a solution should work though.

[MitchDart]

I would like to find out if this was fixed? I have the same issue at the moment. Upstream is not getting my client certificate for gRPC. I tried to go through the PR and it seems like grpc_set_header should automatically work but I can't be sure. Does this mean auth-tls-pass-certificate-to-upstream should just work now?