Make it easier to "attach" a VirtualServerRoute to a VirtualServer without requiring that the VirtualServer be modified

[brianehlert]

There are many customers that like the master-minion patter of the Ingress Resource.
Specifically the ability to define the Master that represents the hostname and core attributes such as TLS and then allow Minions to be defined that modify the behavior of that hostname by defining paths or other behavior modifiers.
This is also called "mergeable ingress".

Four years ago this project introduced its CRDs as an answer to problems that customers had identified with the Ingress resource, specifically the ability to provide safety guarantees around different portions to the configuration. The K8s RBAC allowed an easy way to secure the different objects as a way to provide that safety.

One point of feedback that has come back repeatedly is that the similarity of the VirtualServer and VirtualServerRoute pattern, while powerful, does not offer the simplicity of the mergeable ingress pattern.
The limitation being that a VirtualServerRoute cannot simply and easily extend a VirtualServer object.

The core problem being that the VirtualServer object must be modified whenever a VirtualServerRoute is added. Thus creating both a two step process that is difficult to automate, but also requiring that the person that adds the VSR also have access to the VS (thus breaking the safety boundary) or that two people in different groups have to orchestrate change with each other.

The end result is that it creates a core piece of friction that limits adoption of the VirtualServer/VirtualServerRoute and related CRD objects - which have already proven themselves more powerful and capable than Ingress with its mergeable and annotation pattern.

I am starting this discussion to explore ideas around how we can address this friction point between VS and VSR and still meet the core intention of creating an RBAC boundary and separation of responsibility that promotes configuration safety for those customers that continue to need that.

[brianehlert]

One idea that I had from looking at Gateway API and the pattern of RefernceGrant is that in the VirtualServer, instead of referencing descrete VirtualServerRoutes, would it work if rather a namespace or label were defined.

The intention of using a namespace or label is two fold:

• form an implicit trust with some 'folder' that can be secured
• have a way to inform a watcher that can then monitor that namespace / label for new VirtualServerRoute objects
• automatically attempt to merge those objects into the VirtualServer configuration block of NGINX

[pleshakov]

I would separate ReferenceGrant feature from the problem of selecting which VirtualServerRoute, because it is an additional security feature, rather than a mechanism to define which VSR to select. In fact, we can add ReferenceGrants right now, so it works with existing VS->VSR explicit selection feature, to allow users to prevent their VSRs included into a VS from a namespace they don't trust.

[brianehlert]

ReferenceGrant was suggested as a concept, since as you articulate, this is about making a relationship.
Adding paths without continually modifying the hostname. The flexibility of Ingress master-minion, but with some controls.

[pleshakov]

Here is one possible approach.

Instead of explicitly configuring VSRs by namespace/name in a VSR, introduce a VSR selector, something similar to Gateway API https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.RouteNamespaces

Example

Before

taken from https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/cross-namespace-configuration

VS:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
  namespace: cafe
spec:
  host: cafe.example.com
  tls:
    secret: cafe-secret
  routes:
  - path: /tea
    route: tea/tea
  - path: /coffee
    route: coffee/coffee

Tea VSR

apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
  name: tea
  namespace: tea
spec:
  host: cafe.example.com
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  subroutes:
  - path: /tea
    action:
      pass: tea

Coffee VSR:

apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
  name: coffee
  namespace: coffee
spec:
  host: cafe.example.com
  upstreams:
  - name: coffee
    service: coffee-svc
    port: 80
  subroutes:
  - path: /coffee
    action:
      pass: coffee

In this example, if a user needs to add another VSR with path /juice, it will be necessary to modify the VS to add the corresponding path with the explicit route reference.

New approach

VS:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
  namespace: cafe
spec:
  host: cafe.example.com
  tls:
    secret: cafe-secret
  routes:
  - path: /
     routeSelector: # select VSRs from all namespaces with the label app: cafe
        namespaces: 
          from: Selector
          selector:
            matchLabels:
             app: cafe

Tea VSR - same

Coffee VSR - same

In this example, if a user needs to add another VSR with path /juice, no modifications will be necessary to the VS, as long as the VSR namespace is included in the corresponding selector in the VS.

More details

Personas

Same as before:

• Admin is responsible for defining a VS.
• Users are responsible for VSRs

How does a VSR get attached to a VS

For a VSR to get attached to a VS, the following needs to happen:

• The VS must include a selector that will select that VSR
• The VSR must has the same hostname as the VSR
• The VSR must have the same ingressClassName as the VS, so that the correct IngressController picks the VSR

Note: a VSR will only get attached to a single VS as before. This is ensured by the fact that Ingress Controller only allows one VS per hostname.

Conflicts

Ingress Controller will need to resolve conflicts, because two VSRs might define conflicting rules. For example:
VSR 1

  - path: /path
    action:
      pass: app-1

VSR 2

  - path: /path
    action:
      pass: app-2

The Ingress Controller will handle those conflicts by selecting a winning rule.

Status Information

• Extend status of the VS so that includes how many VSRs it has picked up
• Extend status of the VSR so that includes information if some or all of its routing rules were ignored because they conflict with the another VSR rules.

[brianehlert]

I am assuming the conflict resolution rules can be similar to the current pattern from Ingress for consistency:
https://docs.nginx.com/nginx-ingress-controller/configuration/handling-host-and-listener-collisions/

[vepatel]

VS can use labels to find corresponding VSRs,

[brianehlert]

I think that labels might be similar in thinking to the Gateway RefernceGrant concept.
ReferenceGrant is a type of authorization being granted by the owner of the Gateway object. "Routes that meet this criteria are allowed" is how I think about it.
It is below RBAC. Very similar to my interpretation of the original thinking of the VirtualServer and VirtualServerRoute relationship. The explicit reference in the VirtualServer being the grant. It is however, too specific and thus creates too much friction and does not allow enough flexibility.
The entire opposite of this is the Ingress master-minion - which allows so much flexibility that there are customers who gate the ability to modify, as it is too easy to just throw something in there and break the world. It also has an assumption that everyone is an admin, which also does not fly.

[brianehlert]

Bumping this as it continues to be a requested item.
Something that can be achieved with Ingress and its inheritance (aka master/minion) model but not as flexible for VirtualServer with VirtualServerRoute.

[brianehlert]

There is another discussion that appears related.
#4289
Which reads almost as it could be achieved by @pleshakov 's proposal.

[tstraley]

I'd like to bring the discussion of "path types" into the fold.

As of the current v3.4.3 release, a VSR that uses a regex subroute path type must have an exact matching regex path in the VS it is associated with. This is an additional limitation that would still require updates to the VS for any VSR addition.

I'm curious if, with this proposal, it may be worth considering what the solutions could look like without requiring a VS path to be defined at all, or provide alternative ways of performing validation that a VSR's subroute path is allowed by the VS owner.

[brianehlert]

With Ingress master-minion we are careful about inheritance. And specifically disable capabilities from the mater when they are present in the minion.
This is all about not creating artificial conflicts.
Here, I would expect the relationship to be similar to Ingress RegEx path support today.

[jo-carter]

I would suggest utilizing nested locations in the underlying NGINX configuration in any major rework of VSRs, especially if wildcard selectors of all VSRs across a namespace will be included in that - and assuming paths will still be enforced to some extent.

It will greatly reduce the scope of impact for any misconfigurations (fewer accidental path collisions ect), and will reduce the need for complex validation logic to be layered in (outside of NGINX).

[brianehlert]

Much of this is present for the Ingress resource to support master-minion. There is no reason why we should not follow that established pattern of business logic.

[vepatel]

#4289 has fair bit of over-lap, see #4289 (reply in thread)