Wiring policies with kuadrant backed services

[eguzki]

The Kuadrant merge PR #48 changed how policy controllers are being deployed. However, the code remained mostly the same. And together with the code, the assumptions made initially when the policy controllers were part of the kuadrant instance deployment, not part of the operator’s deployment. The first spotted issue is explained in #69. It was fixed in #109. However, the fix does not cover all issues when the Kuadrant's namespace is not the default one, i.e. kuadrant-system.

Regarding Limitador

The rate limit controller needs to know the address of Limitador's endpoint to configure rate limiting in the gateways accordingly. The Limitador’s service endpoint URL is still provided to the rate limit controller via an environment variable. At boot time. That implies that the Limitador’s service endpoint URL needs to be known at the operator deployment time. Remember that the policy controllers are deployed as part of the operator as describe in #69. In other words, you need to deploy the kuadrant operator with an environment variable saying that kuadrant will be installed in namespace K and then create a Kuadrant CR in the very same K namespace. Clearly, this is an issue and needs to be fixed.

Regarding Authorino

Authorino is a k8s controller and the Kuadrant's operator deploys it in cluster-wide mode
without any sharding defined. When the user creates an auth policy, the auth policy controller does not need to know where authorino lives. The auth policy controller just creates the AuthConfig CR and expects Authorino to read it, even if they are not in the same namespace. There is room for improvement here. The AuthConfig CR managed by Kuadrant should only be used for Authorino instances managed by kuadrant. The cluster could have more authorino instances deployed and kuadrant managed AuthConfig CR's should not be read by other Authorino instances. Istio (or OSSM) needs to be configured with the Authorino endpoint address. But this is one-time registration is done by the kuadrant CRD controller, which is also responsible of deploying Authorino.

The current implementation lacks a required wiring mechanism between policies and Kuadrant backed services. The cluster admin should install the operator (without specifying any namespace for limitador), then install Kuadrant instance in some namespace K. The users of Kuadrant should create policies in their namespaces without knowing anything about the kuadrant backing services. When the controller gets notified about a new policy, the controller should have a way to know about deployed kuadrant's infrastructure (namespaces, services and endpoints) and use it for the the task needed to be done.

Related issues: #56

How it worked before #48

When an instance of Kuadrant, represented by a Kuadrant custom Resource (RC), was created, the following workflow was run by the Kuadrant operator:

• Read the Kuadrant custom resource, paying attention to the namespace. Let's call K the namespace where the Kuadrant CR is created.
• Deploy one Limitador instance in the K namespace
• Deploy one Authorino instance in the K namespace
• Register Authorino instance living in K in the Istio system as an external authorization service.
• Deploy the RateLimitPolicy controller passing as env var the address of the limitador instance in the K namespace.
• Deploy the AuthPolicy controller

When the user created a rate limit policy, the controller already knew about the Limitador's location (name and namespace) to configure it accordingly with the spec of the policy.

[eguzki]

#109 actually implemented a wiring mechanism where the gateway as well as the policies are annotated with kuadrant.io/namespace=NS. The rate limit controller no longer gets limitador endpoint address from an env var.

Using the annotation mechanism, only reading the policy, the controllers know where to find Limitador/Authorino.

This wiring mechanism has room for improvement. But good enough for now, so closing this issue.