[ENH] - Support for pinning the IP address of the load balancer and ability to use internal VPC

[costrouc]

Feature description

For Traefik we currently ask Kubernetes to create a load balancer for us without having any control over the ip that is given or whether it uses an ip address within the current VPC or external (external is default). We have a PR that does this for AWS #650 but never merged it in due to not being totally sure on how we are going to support a feature like this.

For me the conflicting problem is how are we going to support all these options in the qhub-config.yaml and ensure that they are all documented. I'd like to use this issue as a motivation to having a key terraform_overrides which get passed to the terraform module.

The idea would be to have the following in qhub-config.yaml where ingress.terraform_overrides is an arbitrary nested dictionary. This feature is similar to a prior feature we have of passing overrides to helm charts.

ingress:
  terraform_overrides:
      load_balancer_annotations:
          "kubernetes.io/role/internal-elb": 1
      load_balancer_ip: "1.2.3.4"

This will need to be added to the https://github.com/Quansight/qhub/blob/main/qhub/schema.py along with the documentation that this new key is available.

Next we need to make sure that we are passing these overrides to the https://github.com/Quansight/qhub/tree/main/qhub/template/stages/04-kubernetes-ingress stage where ingress is being deployed. This is done via adding to https://github.com/Quansight/qhub/blob/main/qhub/stages/input_vars.py#L144 and passing anything in terraform_overrides as input.

Finally need to add these new variables to https://github.com/Quansight/qhub/blob/main/qhub/template/stages/04-kubernetes-ingress/variables.tf and properly pass in the values to https://github.com/Quansight/qhub/blob/main/qhub/template/stages/04-kubernetes-ingress/modules/kubernetes/ingress/main.tf#L58.

Value and/or benefit

Enterprise support often requires deploying within an existing VPC and pinnning the ip address.

Anything else?

No response

[costrouc]

Also partially implemented in https://github.com/Quansight/qhub/compare/azure_extension

[aktech]

Working on this for GCP, here are some notes (from GKE docs) :

• By default, GKE will deploy an internal TCP/UDP load balancer using the node subnet range
• The subnet can be user-specified on a per-Service basis using the networking.gke.io/internal-load-balancer-subnet annotation.
• The subnet must exist before it is referenced by the Service.

By default an available IP from the subnet will automatically be chosen but we can also specify the loadBalancerIP but it must be part of the referenced subnet.

Annotations required:

• networking.gke.io/load-balancer-type: "Internal"
• networking.gke.io/internal-load-balancer-subnet: "pre-existing-subnet"

This means: - The subnet must also be in the same VPC and region as the GKE cluster.

References:

• https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
• https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing
• https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing#ilb_psc

[dhirschfeld]

In case it's useful, for my AKS traefik deployment I use*:

  service:
    enabled: true
    type: LoadBalancer
    spec:
      loadBalancerIP: "1.2.3.4"
      externalTrafficPolicy: Local
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

• https://docs.microsoft.com/en-us/azure/aks/internal-lb#create-an-internal-load-balancer
• https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer

Note: the externalTrafficPolicy is there to allow traefik to log / pass on the real client IP.

---

* That was my old config. I now use Azure DNS to point to the location of the traefik pod, wherever it is spun up in the AKS subnet.

[aktech]

Thanks @dhirschfeld that's really helpful.