Support cross-namespace BackendRefs in HTTPRoutes (#806)

Allow HTTPRoutes to reference Backends in different namespaces if a ReferenceGrant permits it. Problem: NKG does not allow HTTPRoutes to reference Backends in different namespaces. Solution: Allow HTTPRoutes to reference Backends in different namespaces if a ReferenceGrant permits it. Both SecretObjectReferences and BackendObjectReferences are resolved through the referenceGrantResolver. The referenceGrantResolver transforms the map of ReferenceGrants into a structure that allows for O(n) lookup of references.
nginxinc · Jul 6, 2023 · 219c973 · 219c973
1 parent 8ca16d2
commit 219c973
Show file tree

Hide file tree

Showing 18 changed files with 760 additions and 243 deletions.
diff --git a/docs/gateway-api-compatibility.md b/docs/gateway-api-compatibility.md
@@ -12,7 +12,7 @@ This document describes which Gateway API resources NGINX Kubernetes Gateway sup
 | [TLSRoute](#tlsroute) | Not supported |
 | [TCPRoute](#tcproute) | Not supported |
 | [UDPRoute](#udproute) | Not supported |
-| [ReferenceGrant](#referencegrant) |  Partially supported |
+| [ReferenceGrant](#referencegrant) |  Supported |
 | [Custom policies](#custom-policies) | Not supported |
 
 ## Terminology
@@ -152,19 +152,18 @@ Fields:
 
 ### ReferenceGrant
 
-> Status: Partially supported.
-
-NKG only supports ReferenceGrants that permit Gateways to reference Secrets. 
+> Status: Supported.
+> Support Level: Core
 
 Fields:
 * `spec`
   * `to`
     * `group` - supported.
-    * `kind` - partially supported. Only `Secret`.
+    * `kind` - supports `Secret` and `Service`.
     * `name`- supported.
   * `from`
     * `group` - supported.
-    * `kind` - partially supported. Only `Gateway`.
+    * `kind` - supports `Gateway` and `HTTPRoute`.
     * `namespace`- supported.
 
 ### Custom Policies

diff --git a/examples/cross-namespace-routing/README.md b/examples/cross-namespace-routing/README.md
@@ -0,0 +1,128 @@
+# Example
+
+In this example, we expand on the simple [cafe-example](../cafe-example) by using a ReferenceGrant to route to backends
+in a different namespace from our HTTPRoutes.
+
+## Running the Example
+
+## 1. Deploy NGINX Kubernetes Gateway
+
+1. Follow the [installation instructions](/docs/installation.md) to deploy NGINX Gateway.
+
+1. Save the public IP address of NGINX Kubernetes Gateway into a shell variable:
+
+   ```
+   GW_IP=XXX.YYY.ZZZ.III
+   ```
+
+1. Save the port of NGINX Kubernetes Gateway:
+
+   ```
+   GW_PORT=<port number>
+   ```
+
+## 2. Deploy the Cafe Application
+
+1. Create the cafe namespace and cafe application:
+
+   ```
+   kubectl apply -f cafe-ns-and-app.yaml
+   ```
+
+1. Check that the Pods are running in the `cafe` namespace:
+
+   ```
+   kubectl -n cafe get pods
+   NAME                      READY   STATUS    RESTARTS   AGE
+   coffee-6f4b79b975-2sb28   1/1     Running   0          12s
+   tea-6fb46d899f-fm7zr      1/1     Running   0          12s
+   ```
+
+## 3. Configure Routing
+
+1. Create the `Gateway`:
+
+   ```
+   kubectl apply -f gateway.yaml
+   ```
+
+1. Create the `HTTPRoute` resources:
+
+   ```
+   kubectl apply -f cafe-routes.yaml
+   ```
+1. Create the `ReferenceGrant`:
+
+   ```
+   kubectl apply -f reference-grant.yaml
+   ```
+   This ReferenceGrant allows all HTTPRoutes in the `default` namespace to reference all Services in the `cafe`
+   namespace.
+
+## 4. Test the Application
+
+To access the application, we will use `curl` to send requests to the `coffee` and `tea` Services.
+
+To get coffee:
+
+```
+curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
+Server address: 10.12.0.18:80
+Server name: coffee-7586895968-r26zn
+```
+
+To get tea:
+
+```
+curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea
+Server address: 10.12.0.19:80
+Server name: tea-7cd44fcb4d-xfw2x
+```
+
+## 5. Remove the ReferenceGrant
+
+To restrict access to Services in the `cafe` Namespace, we can delete the ReferenceGrant we created in
+Step 3:
+
+```
+kubectl delete -f reference-grant.yaml
+```
+
+Now, if we try to access the application over HTTP, we will get an internal server error:
+```
+curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea
+
+<html>
+<head><title>500 Internal Server Error</title></head>
+<body>
+<center><h1>500 Internal Server Error</h1></center>
+<hr><center>nginx/1.25.1</center>
+</body>
+</html>
+```
+
+You can also check the conditions of the HTTPRoutes `coffee` and `tea` to verify that the reference is not permitted:
+
+```
+kubectl describe httproute coffee
+
+Condtions:
+      Message:               Backend ref to Service cafe/coffee not permitted by any ReferenceGrant
+      Observed Generation:   1
+      Reason:                RefNotPermitted
+      Status:                False
+      Type:                  ResolvedRefs
+      Controller Name:       k8s-gateway.nginx.org/nginx-gateway-controller
+```
+
+```
+kubectl describe httproute tea
+
+Condtions:
+      Message:               Backend ref to Service cafe/tea not permitted by any ReferenceGrant
+      Observed Generation:   1
+      Reason:                RefNotPermitted
+      Status:                False
+      Type:                  ResolvedRefs
+      Controller Name:       k8s-gateway.nginx.org/nginx-gateway-controller
+```
diff --git a/examples/cross-namespace-routing/cafe-ns-and-app.yaml b/examples/cross-namespace-routing/cafe-ns-and-app.yaml
@@ -0,0 +1,74 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cafe
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: coffee
+  namespace: cafe
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: coffee
+  template:
+    metadata:
+      labels:
+        app: coffee
+    spec:
+      containers:
+      - name: coffee
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: coffee
+  namespace: cafe
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: coffee
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tea
+  namespace: cafe
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: tea
+  template:
+    metadata:
+      labels:
+        app: tea
+    spec:
+      containers:
+      - name: tea
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: tea
+  namespace: cafe
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: tea
diff --git a/examples/cross-namespace-routing/cafe-routes.yaml b/examples/cross-namespace-routing/cafe-routes.yaml
@@ -0,0 +1,39 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: coffee
+spec:
+  parentRefs:
+  - name: gateway
+    sectionName: http
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /coffee
+    backendRefs:
+    - name: coffee
+      namespace: cafe
+      port: 80
+---
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: tea
+spec:
+  parentRefs:
+  - name: gateway
+    sectionName: http
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: Exact
+        value: /tea
+    backendRefs:
+    - name: tea
+      namespace: cafe
+      port: 80
diff --git a/examples/cross-namespace-routing/gateway.yaml b/examples/cross-namespace-routing/gateway.yaml
@@ -0,0 +1,13 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+  name: gateway
+  labels:
+    domain: k8s-gateway.nginx.org
+spec:
+  gatewayClassName: nginx
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
+    hostname: "*.example.com"
diff --git a/examples/cross-namespace-routing/reference-grant.yaml b/examples/cross-namespace-routing/reference-grant.yaml
@@ -0,0 +1,13 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: access-to-cafe-services
+  namespace: cafe
+spec:
+  to:
+  - group: ""
+    kind: Service
+  from:
+  - group: gateway.networking.k8s.io
+    kind: HTTPRoute
+    namespace: default
diff --git a/examples/https-termination/README.md b/examples/https-termination/README.md
@@ -164,6 +164,8 @@ curl: (7) Failed to connect to cafe.example.com port 443 after 0 ms: Connection
 You can also check the conditions of the Gateway `https` Listener to verify the that the reference is not permitted:
 
 ```
+ kubectl describe gateway gateway
+
  Name:                    https
  Conditions:
    Last Transition Time:  2023-06-26T20:23:56Z

diff --git a/examples/https-termination/reference-grant.yaml b/examples/https-termination/reference-grant.yaml
@@ -1,7 +1,7 @@
 apiVersion: gateway.networking.k8s.io/v1beta1
 kind: ReferenceGrant
 metadata:
-  name: allow-default-to-cafe-secret
+  name: access-to-cafe-secret
   namespace: certificate
 spec:
   to: