KEP for decoupling KIC components

[shaneutt]

What this PR does / why we need it:

Previously we had considered KEP 1 declined due to other higher precedent priorities, but now as there are multiple related and synergistic features in flight such as the imminent preview release of Gateway support and the Kong Kubernetes Team considering a plan for what a v1 Operator release might look like and we're also getting recent updates on old (but still valid) issues such as #702 and #986 now seems like a good time to revamp and reconsider how we resolve some of the traditionally tightly coupled components in KIC.

[rainest]

Per #986 (comment) I'm skeptical of the benefit of separating the webhook. You'll still need to update it, and those updates still have the same outage characteristics, so we're effectively just moving the problem elsewhere. IMO a default PDB makes more sense, but I'm not sure if a MinAvailable of 1 would play nicely with our default single-replica Deployments. Brief research suggests the answer is "it doesn't", so we'd probably need to make that chart-only with a conditional MinAvailable if the replica count >= 2. This would probably be a breaking change due to our existing PDB configuration.

One->many DB-less instance management needs to address the security concerns with exposing the admin API in the simplest implementation, as there's no standard way to secure a DB-less admin API listen. I think we'd probably need to create a Golang implementation of the hybrid mode protocol and make the controller an imitation control plane node so we can benefit from its built-in mTLS requirement.

[seh]

there's no standard way to secure a DB-less admin API listen

Are you ruling out NetworkPolicy to handle that? You could reject ingress to the Admin port from pods other than the ingress controllers.

[rainest]

NetworkPolicy should be viable, but it looks like it's off by default in the major providers (with bonus "bring your own CNI stack" on EKS):

https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy#enabling_network_policy_enforcement
https://aws.github.io/aws-eks-best-practices/security/docs/network/#network-policy
https://docs.microsoft.com/en-us/azure/aks/use-network-policies#create-an-aks-cluster-and-enable-network-policy

[seh]

it looks like it's off by default in the major providers

Yes, that's sad. We use it to good effect with Calico in our AWS-hosted clusters. In GKE, well, we have plans. We're not using the "V2 dataplane" yet.

[shaneutt]

Given the contention about the webhook I'm going to drop this change in favor of keeping that as a separate concern so it doesn't overwhelm the conversation.

#2256 is the replacement, and the webhook stuff will be considered separately.