Control plane Ingress controller is tied to Bootstrap

[gdemonet]

Component: salt, ingress

Problem:

The way the control-plane Ingress controller is currently deployed ain't great (using an externalIP).
(see this comment)

This means the controller can only run on the Node with this externalIP value, in our case, the Bootstrap. (Note that the certificates are also only generated for this IP in their SANs)

Resolution proposal:

This should be fixed somehow anyway (basically, your control-plane ingress running on a non-bootstrap node won't even work today). Either by having a specific portmap configuration, or by running them in the host network namespace and bind to the (host) control-plane IP only, somehow.
(see this comment)

[TeddyAndrieux]

Exploration result:

Have a control plane ingress working on all nodes is not difficult with today's approach, we just have to add all control plane IPs as externalIPs in the ingress Service, but we are still stuck to bootstrap for all redirectURL used for authentication, ect. (So if you lose the bootstrap node you still lose access to the UIs.

---

Several approaches to solves these issues (list likely, not exhaustive):

1- Use an FQDN everywhere as "redirectURL/URI" so that we let the DNS server resolve the name to a working server IP

NOTE: this approach still need to configure all control plane IPs as externalIPs

Cost: ~2-3 days
Pros:

• simple to do
• "HA" not managed on MetalK8s side but on the user environment

Cons:

• "Fake HA", needs an active DNS to be really HA

2- Add some optional VIP management in MetalK8s (using keepalived or something like this), so that we can use this VIP everywhere, this keepalived management is optional and you can provide whatever you want as "control plane endpoint" (including an FQDN)

NOTE: this approach do not need all control plane IPs as externalIPs we only use the VIP one

Cost: ~5 days
Pros:

• Real HA

Cons:

• It's likely not possible in every user environment (that's why it's optional)
  NOTE: When it's not possible, the user will need to manage the VIP himself manually, or use an FQDN and basically do the same as "1-"

3- Use relative path everywhere so that we just use always stick to the IP used to reach the UI initially

NOTE: Just did some simple tests and it does not seem to work as is (it's maybe possible with some "tools" but I'm not sure)
NOTE: this approach still need to configure all control plane IPs as externalIPs

Cost: ?
Pros:

• "HA" not managed on MetalK8s side
• Simple HA, just need load balancing on control plane IPs

Cons:

• (Not sure it's possible)

---

Some other approaches are possible (using MetalLB maybe ? or some other tools).

IMHO, the solution "2-" is the one we should do now, it's really flexible and not really costly

[TeddyAndrieux]

MetalLB seems good for control plane ingress but it still needs to be optional as it's not possible in every environment. So will need some new entry in Bootstrap configuration file for it

NOTE: We really want this to be configurable at any point (basically edit the bootstrap config, re-run "this" salt state, and Go)

If we use MetalLB as "LoadBalancer" for Control Plane in MetalK8s, we do no longer need to use DaemonSet nginx ingress and we can go with a Deployment (with 2 replicas, likely)

---

POC MetalLB:

• MetalLB installed with "basic" manifests (with some tuning to only run on control plane nodes)
• Simple MetalLB configuration
• Created a LoadBalancer service for control plane ingress with an explicit loadBalancerIP and externalTrafficPolicy: Local

IP switch really fast when a node goes down and sends proper "gratuitous ARP" when switching

NOTE: From MetalLB doc we cannot share IP if we use Local external traffic policy https://metallb.universe.tf/usage/#ip-address-sharing