adding support for secrets for backendtlspolicy

[porthorian]

Proposed changes

The proposed changes here adds the support for Secrets to be used according to the BackednTLSPolicy Custom Resource. This helps further the implementation of nginx-gateway-fabric to support the Gateway API more.

Problem

Currently BackendTLSPolicy only supports readying in tls certifications and ca certifications via a config map. This does not work when you are using cert-manager for instance since it puts this information into a kubernetes secret rather then a config map.

Solution

The Solution here is to hook into the existing Configuration structure that builds the dataplane nginx configuration that is served in the pod. Allowing for secrets and configmaps to be read into this array of CertBundles.

Testing

Did some end to end testing here and built 2 packages that can be used with this code.

https://github.com/porthorian/nginx-gateway-fabric/pkgs/container/nginx-gateway-fabric%2Fnginx
https://github.com/porthorian/nginx-gateway-fabric/pkgs/container/nginx-gateway-fabric

These were then deployed and made a BackendTLSPolicy with the secret that was created with cert-manager.

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: tls-upstream-unifi
spec:
  targetRefs:
    - kind: Service
      name: unifi
      group: ""
  validation:
    caCertificateRefs:
      - name: unifi-signed-cert
        group: ''
        kind: Secret
    hostname: {your-domain-name}

—

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: self-signed-svc-cert
spec:
  dnsNames:
    - {your-domain-name}
  secretName: unifi-signed-cert
  commonName: unifi
  issuerRef:
    name: self-signed-ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io
  keystores:
    jks:
      alias: unifi
      create: true
      passwordSecretRef:
        name: unifi-keystore
        key: password

Reviewer Focus

Since this is a working draft here - places that should be focused on is making sure I am inline with the current standards and everyone is statisified with how this was structurally laid out before moving onto polishing it up and writing some additional tests.

Issues

Closes #2629

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

* Adds support for secrets to be used in BackendTLSPolicy Gateway API for tls certificates and ca certificates.

[sjberman]

Overall I think the approach looks good here! Haven't yet tested it myself, but will get around to that. Thanks for this.

[salonichf5]

Steps used to test the PR

1. Create a self-signed issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

2. Create a cert with a secret

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: backend-tls-cert
  namespace: default  # Adjust based on your setup
spec:
  secretName: backend-tls-secret
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days before expiration
  subject:
    organizations:
      - "My Organization"
  isCA: true
  privateKey:
    algorithm: RSA
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
    - cafe.example.com  # Replace with your backend domain
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer

3. Create a backend TLS Policy referencing the secret created above

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: tls-upstream-unifi
spec:
  targetRefs:
    - kind: Service
      name: coffee
      group: ""
  validation:
    caCertificateRefs:
      - name: backend-tls-secret
        group: ''
        kind: Secret
    hostname: cafe.example.com

4. Create a gateway using TLS mode and secret created

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP

5. Create a svc and deployment using the https listener

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
spec:
  replicas: 1
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
        - name: coffee
          image: nginxinc/nginx-unprivileged:latest
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: secret
              mountPath: /etc/nginx/ssl
              readOnly: true
            - name: config-volume
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: secret
          secret:
            secretName: backend-tls-secret.  <-- use the secret name created
        - name: config-volume
          configMap:
            name: secure-config
---
apiVersion: v1
kind: Service
metadata:
  name: coffee
spec:
  ports:
    - port: 8443
      targetPort: 8443
      protocol: TCP
      name: https
  selector:
    app: coffee
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-config
data:
  app.conf: |-
    server {
      listen 8443 ssl;
      listen [::]:8443 ssl;

      server_name cafe.example.com;

      ssl_certificate /etc/nginx/ssl/tls.crt;
      ssl_certificate_key /etc/nginx/ssl/tls.key;

      default_type text/plain;

      location / {
        return 200 "hello from pod coffee\n";
      }
    } 

6. Create a route referencing the this gateway

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: coffee
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - "cafe.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    backendRefs:
    - name: coffee
      port: 8443

7. Test using curl matching against the cert in backendTLSPolicy

curl -v --resolve cafe.example.com:8443:$GW_IP https://cafe.example.com:8443/coffee --cacert <(kubectl get secret backend-tls-secret -n default -o jsonpath='{.data.ca\.crt}' | base64 -d)

* Added cafe.example.com:8443:127.0.0.1 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 127.0.0.1:8443...
* Connected to cafe.example.com (127.0.0.1) port 8443
Handling connection for 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /dev/fd/11
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: O=My Organization
*  start date: Feb 11 17:24:32 2025 GMT
*  expire date: May 12 17:24:32 2025 GMT
*  subjectAltName: host "cafe.example.com" matched cert's "cafe.example.com"
*  issuer: O=My Organization
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://cafe.example.com:8443/coffee
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: cafe.example.com:8443]
* [HTTP/2] [1] [:path: /coffee]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /coffee HTTP/2
> Host: cafe.example.com:8443
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 502
< server: nginx
< date: Tue, 11 Feb 2025 18:07:46 GMT
< content-type: text/html
< content-length: 150
<
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host cafe.example.com left intact

The cert matches but running into Bad Gateway probably due to something in my setup. Hence, posting it here for more info

Any insights - @sjberman

[sjberman]

@salonichf5 I don't think you need to use this Secret at all at the Gateway layer. This is just for securing traffic to the backend, so you should just be able to use http for client -> nginx traffic.

https://docs.nginx.com/nginx-gateway-fabric/how-to/traffic-security/securing-backend-traffic/

[salonichf5]

Verifies that the PR is working as expected.

Nginx config for cafe.example.com

server {
    server_name cafe.example.com;
    location /coffee/ {
        proxy_ssl_server_name on;
        proxy_ssl_verify on;
        proxy_ssl_name cafe.example.com;
        proxy_ssl_trusted_certificate /etc/nginx/secrets/cert_bundle_default_backend-tls-secret.crt;
    }
    location = /coffee {
        proxy_ssl_server_name on;
        proxy_ssl_verify on;
        proxy_ssl_name cafe.example.com;
        proxy_ssl_trusted_certificate /etc/nginx/secrets/cert_bundle_default_backend-tls-secret.crt;
    }

curl -v  --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
> GET /coffee HTTP/1.1
> Host: cafe.example.com:8080
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
Handling connection for 8080
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 11 Feb 2025 20:08:39 GMT
< Content-Type: text/plain
< Content-Length: 22
< Connection: keep-alive
<
hello from pod coffee

Verified that the certificate contents in backend-tls-secret match the mounted certificate in /etc/nginx/secrets/cert_bundle_default_backend-tls-secret.crt

[salonichf5]

Looks good to me once we have issues with the linter fixed!

[salonichf5]

Linter still fails, you can verify what's happening locally by running make lint from the root directory. Let me know once you are ready and I can re-trigger the pipeline. @porthorian

[sjberman]

@salonichf5 It may be related to a recent fix in the linter which has been merged to main. Rebasing could help fix the issue.