Multiple kuadrant instances #5
Conversation
| * One Authorino deployment instance | ||
| * A list of dedicated gateways. Those gateways cannot be shared between multiple kuadrant instances. | ||
|
|
||
|  |
There was a problem hiding this comment.
The diagram can be seen also from the branch: https://github.com/Kuadrant/architecture/blob/wire-policies-with-backing-service/rfcs/0000-multiple-kuadrant-instances.md
eguzki
changed the title
|
looking for review @Kuadrant/engineering |
|
|
||
| When the policy's `targetRef` targets a Gateway, there is a direct reference to the gateway. | ||
|
|
||
| When the policy's `targetRef` targets an HTTPRoute, Kuadrant will follow the [`parentRef`](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io%2fv1beta1.CommonRouteSpec) attribute which should be a direct reference to the gateway or gateways. |
There was a problem hiding this comment.
This means that indirectly, a KAP or RLP, would be applied to more than 1 Kuadrant instance. As mentioned before, the control plane in charge of associating policy with 1 Kuadrant instance, will need to decide which policy apply in case there's another HTTPRoute sharing one or more of parentRef gateways
There was a problem hiding this comment.
OK, it's mentioned that this use case is not supported , still we might need to provide a way to mitigate since one might not have access to modify the network topology
| * Add annotation in the gateway with a value of the Name/Namespace of the Kuadrant CR. | ||
| * This approach is commonly used. Requires annotation management. |
There was a problem hiding this comment.
Even tho this would require the extra management, might be the simpler and more flexible way. At the moment, the control plane assumes there's only one Kuadrant instance and annotates every single gateway. Both the Kuadrant CR and Gateways targeted should be in sync.
There was a problem hiding this comment.
We could define the list of gateways that meant to be managed in the Kuadrant CR.
There was a problem hiding this comment.
I'd suggest a list of GatewayClasses instead of Gateways.
Users can always "split" one GatewayClass into two or more, handled by the same controller or not, to highlight the fact that different configurations apply to each class. In fact, this is encouraged by GW-API:
We expect that one or more GatewayClasses will be created by the infrastructure provider for the user. It allows decoupling of which mechanism (e.g. controller) implements the Gateways from the user. For instance, an infrastructure provider may create two GatewayClasses named internet and private to reflect Gateways that define Internet-facing vs private, internal applications.
Some discussion about this also here: #7 (comment)
There was a problem hiding this comment.
Kuadrant manages gateway instances, not gateway classes. Kuadrant instance A may manage gateway X and instance B may mananage instance Y. Both X and Y gateways may be share the same gateway class.
There was a problem hiding this comment.
I guess what I'm saying is that Kuadrant can use the GatewayClass to know which Gateways to manage.
In your provided example,
Kuadrant instance A may manage gateway X and instance B may mananage instance Y
- You can have both instances of Kuadrant listing the one and only GatewayClass; or
- You can have 2 GatewayClasses, one for each gateway, one for each instance of Kuadrant.
| * Protection against external traffic load spikes. Other tenant's traffic spikes does not affect Authorino/Limitador throughput and delay as it would when shared. | ||
| * No need to have cluster administrator role to deploy a kuadrant instance. One tenant administrator can manage gateways, Limitador and Authorino instances (including deployment modes). | ||
| * The cluster administrator gets control and visibility across all the Kuadrant instances, while the tenant administrator only gets control over their specific gateway(s), Limitador and Authorino instances. | ||
| * (looking for ideas for more benefits)... |
There was a problem hiding this comment.
I think the key benefits are protection against noisy neighbour, isolation particularly in the auth context, and the ability to independently scale based on usage.
|
|
||
|  | ||
|
|
||
| The main reason is related to the rate limiting capability. The limits specified in the RateLimit Policy would be enforced per kuadrant instance basis (provided by Limitador instance). Thus, traffic hitting one gateway would see different rate limiting counters than traffic hitting the other gateway. The user would expect X rps and actually it would be X rps per gateway. For consistency reasons, when this configuration happens, the control plane will reject the policy. |
There was a problem hiding this comment.
While I think this is a good first step and we indeed should reject such policies initially, I think there might be a way to support this as we get to architect for multiple cluster support. The use-case isn't much different. What I'm thinking is that each limitador would know about each other. Upon hitting a "shared" limit, they'd start sharing the counters for it. Which, other than latency being lower and cross communication being easier, there isn't much differences between the two applications.
Sharing this thought here as an FYI and sharing a possibly path forward. Again, it looks perfectly acceptable to have that limitation initially.
|
|
||
| TODO | ||
|
|
||
| # Unresolved questions |
There was a problem hiding this comment.
It might already be work in progress, but a specification of how errors (and which error cases) would be surfaced to the user would be good. For instance, a policy is applied to a HTTPRoute alright. The route gets an additional Gateway wired to it… what's the behavior? How does the user know?
There was a problem hiding this comment.
a policy is applied to a
HTTPRoutealright. The route gets an additionalGatewaywired to it… what's the behavior? How does the user know?
Currently, the addition of a new Gateway to a HTTPRoute targeted by a policy is not reflected in the policy status, just like any other event of modifying the HTTPRoute that is handled successfully by the controller. The controller basically only reports "HTTPRoute is protected" or reconciliation errors. But even with the error messages, they are not always helpful to users.
I think this is already important today and even more so in a context of multiple Kuadrant instances per cluster. Users could benefit from the info of which Gateways have been configured for a policy, which were expected to but have not (and why not), in the policy status. Currently, users can only partially work that out by reading in the annotations of the Gateways which policies affect the Gateway.
There was a problem hiding this comment.
| gatewaysSelector: | ||
| matchLabels: | ||
| app: kuadrant-a |
There was a problem hiding this comment.
I've been thinking about the label selector approach for this application lately. Although it is consistent with other usages such as how Authorino selects AuthConfigs, how AuthConfigs select API key and X.509 cert issuer Secrets, how Istio selects workloads, among others, I think for this particular case of selecting (assigning) Gateways for a Kuadrant instance nevertheless an approach based on the GatewayClasses could be a better fit. A few advantages I see:
- More assertive as the Kuadrant CR would list GatewayClasses by name explicitly;
- Rational topology with individual GatewayClasses sometimes split into two or more, making it clear which groups of Gateways are configured by Kuadrant and which are not;
- Easier to validate that the Gateways assigned to a Kuadrant instance are implemented by a compatible gateway provider (Istio, OSSM, EG, etc), with less (or zero) variations of providers within each group of gateways assigned to a Kuadrant instance. (See also Multiple gateway providers in the same cluster #7)
|
out of scope as of today. Maybe some other day |
I would like to reopen this please and clarify first. What is out of scope? The mechanism to filter specific gateways to be in the scope of a Kuadrant instance or supporting multiple gateway providers in the same cluster? I believe the latter should still be in consideration. |
|
This is issue is about kuadrant control plane supporting multiple Kuadrant CR's and the implications of that (related to map gateways to kuadrant instances). |
Roger that, @eguzki. Thanks for explaining. Closing it back then. |
No description provided.