[ci.jenkins.io] Create private EKS cluster with "side" services (datadog, ACP, etc.)

[dduportal]

We need a private EKS cluster to run ci.jenkins.io container agents.

• No need for multiple AZs unless strictly necessary
• We want to use a private endpoint for ci.jenkins.io access to the cluster: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
  • But we also need the public endpoint to be restricted to our VPN (for admin access) and infra.ci.jenkins.io outbound IP (for terraform and Kubernetes management)
• The subnets can (and should) be private (see [ci.jenkins.io] Define virtual networking for AWS #4320)
• EKS networks considerations: https://docs.aws.amazon.com/eks/latest/userguide/network-reqs.html
• Our former Terraform code, using a module: https://github.com/jenkins-infra/aws/blob/ac02157bb6646918594827b293320f64eacffeb5/cik8s-cluster.tf
• Some addons are needed (ref. https://docs.aws.amazon.com/eks/latest/userguide/workloads-add-ons-available-eks.html):
  • Kube Proxy (of course)
  • CoreDNS (of course)
  • VPC CNI to get native VPC networking for pods and services
  • EBS CSI to ensure we can have volumes for Artifact Caching Proxy
• We need the autoscaler. Worth checking Karpenter.
• We also need the AWS LB Controller installed to allow creating a private LB for the Artifact Caching Proxy to ensure VM agents of ci.jenkins.io can reach the service
  • See https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-container-applications-privately-on-amazon-eks-using-aws-privatelink-and-a-network-load-balancer.html for a guide on how to expose privately the ACP service with a private LB
• About Node pools:
  • "application" ARM64 Node pool to host ACP, datadog an any other "non Jenkins agent" pods. Check the current one in azure: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L69
  • Set up Karpenter (see https://karpenter.sh/v0.32/concepts/) to handle builds Node pools (Linux x86, BOM Linux x86, and future Windows + Linux arm64)
  • "agents" x86 Node pool to host usual Linux agents: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L103
  • "bom agents" x86 Node pool to host BOm agents: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L137
  • We should start with no Spot agents for the first months (we have credits and we want to avoid too much build retries
  • Note: see [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS #4318 for upcoming Windows Node pool agents
• We'll have to add this cluster to jenkins-infra/kubernetes-management as usual:
  • An admin SVC account is needed like https://github.com/jenkins-infra/azure/blob/main/ci.jenkins.io-kubernetes-agents.tf#L180-L189
  • Then set up the kubeconfig credential once in SOPS secrts and create the cluster in kubernetes-management
  • Proceed to helmfile releases definitions
  • set up ACP to use a private LB and check access from the ci.jio controller or a VM agent through the private endpoint

[dduportal]

Discussed with @smerle33:

• [ci.jenkins.io] Define virtual networking for AWS #4320 reached a first milestone allowing starting work on the issue here
• Stephane prefers more the EKS topic than the VM topic: assigning EKS to @smerle33
• Scheduling for next milestone when he's back from holidays

[smerle33]

change of usage for the module since last time we used it https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-20.0.md

[smerle33]

We choose to deal with all the IAM usage within the private repository https://github.com/jenkins-infra/terraform-states/commit/cfd08c45dd4153d676c9223670f927d515585679
instead of giving the module user too much power.

[dduportal]

First set of working hypothesis for the initial deployment: Internal SVC and EBS persistence

ACP is now installed (jenkins-infra/terraform-aws-sponsorship#74, jenkins-infra/terraform-aws-sponsorship#75, jenkins-infra/kubernetes-management#6073)

[dduportal]

Next steps (all elements have the same priority):

• Setup Jenkins namespace + SVC account + RBAC, and set up the controller VM IAM identity to map to this SVC account in order to have a credential-less setup (like it has been been with EC2 agents)
  • Don't forget to remove the kubeconfig's output from terraform once setup. Implies eventually updating the helm chart to NOT create a token (unneeded anymore)
• Setup ACP with an internal LB to allow reaching through EC2 agents
  • See https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-container-applications-privately-on-amazon-eks-using-aws-privatelink-and-a-network-load-balancer.html
  • Need to install the EKS Add-on LB (with its own IRSA identity) first
  • Then check the Kubernetes SVC required annotations and AWS Private Link addons

[smerle33]

for part 1 (jenkins namespace, service account, rbac and iam link with VM iam identity and kubernetes service account):

namespace, service account and rbac are dealt with the helm chart: https://github.com/jenkins-infra/helm-charts/tree/main/charts/jenkins-kubernetes-agents

[dduportal]

for part 1 (jenkins namespace, service account, rbac and iam link with VM iam identity and kubernetes service account):

namespace, service account and rbac are dealt with the helm chart: https://github.com/jenkins-infra/helm-charts/tree/main/charts/jenkins-kubernetes-agents

Continuing the Jenkins container Agents related tasks in #4317 (comment)

[dduportal]

Update about the "private link" (e.g. allowing VM agents in other private subnets to access ACP):

As per https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html#network-load-balancer:

You can launch Network Load Balancers in any subnet in your cluster’s VPC, including subnets that weren’t specified when you created your cluster.

=> It means we only need a LB set to internal and associated to the VM agents subnets in the EKS cluster to allow access. No need for private link which is more a "VPC to VPC" tool.

Next steps:

• Install the AWS LB controller (It is an EKS add on which requires an IRSA profile see https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/deploy/installation/)

  • feat(eks/ci.jio-agents-1) install the AWS LB controller (Helm chart which requires an IRSA role) terraform-aws-sponsorship#89
  • feat(eks/ci.jenkins.io-agents2) setup requirement for AWS LB and Artifact Caching Proxy terraform-aws-sponsorship#90

• Set up ACP Kubernetes Service to map to an AWS internal LB in the VM agents subnet

  • We want an IP target (e.g. sending request directly to Pod IPs) instead of node targets (which implies NodePort) as we have CNI setup. See https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html#network-load-balancer
  • feat(ci.jio-agents-2) enable internal LB for ACP kubernetes-management#6113

• Determine the DNS names for both LB (for VMs) and internal EKS (for containers) and verify we can reach ACP in HTTP through both:

  $ curl -I http://k8s-artifact-artifact-51a40b09ac-8673e1bdefab0d64.elb.us-east-2.amazonaws.com:8080
  HTTP/1.1 200 
  Date: Fri, 17 Jan 2025 18:44:53 GMT
  Content-Type: text/html
  Connection: keep-alive
  X-JFrog-Version: Artifactory/7.103.0 710300900
  X-Artifactory-Id: 1b0dde6bb85d6af16009ba7b901d175a8b2ec79a
  X-Artifactory-Node-Id: jenkinsci-artifactory-primary-1
  Access-Control-Allow-Methods: GET, POST, DELETE, PUT
  Access-Control-Allow-Headers: X-Requested-With, Content-Type, X-Codingpedia
  Cache-Control: no-store
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  X-Request-ID: ff0f25a3f7ec4cbf8591b3645e9d50e9:ff0f25a3f7ec4cbf8591b3645e9d50e9:ff0f25a3f7ec4cbf8591b3645e9d50e9:0
  X-Cache-Status: MISS

  $ curl -I http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/
  HTTP/1.1 200 
  Date: Fri, 17 Jan 2025 18:47:16 GMT
  Content-Type: text/html
  Connection: keep-alive
  X-JFrog-Version: Artifactory/7.103.0 710300900
  X-Artifactory-Id: 302b5b145a93f620323b96ee3abb18b071de00d9
  X-Artifactory-Node-Id: jenkinsci-artifactory-primary-0
  Access-Control-Allow-Methods: GET, POST, DELETE, PUT
  Access-Control-Allow-Headers: X-Requested-With, Content-Type, X-Codingpedia
  Cache-Control: no-store
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  X-Request-ID: 372a60b1f394cf32ae8c114778765184:372a60b1f394cf32ae8c114778765184:372a60b1f394cf32ae8c114778765184:0
  X-Cache-Status: MISS

• Update Puppet configuration to add the new ACP URLs (aws-eks-internal and aws-internal) - feat(aws.ci.jenkins.io) set up ACP jenkins-infra#3827

[dduportal]

Update: edited body of the issue to specify we want to use Karpenter (https://karpenter.sh/v0.32/concepts/)

[dduportal]

Update: edited body of the issue to specify we want to use Karpenter (https://karpenter.sh/v0.32/concepts/)

Last step: Remove cluster autoscaler and set up Karpenter instead.

Rationale of using Karpenter instead of Cluster Autoscaler: instead of artificially constraining nodes to some size, we can relax these constraint to always get a node at the right price.
It also makes node groups and node management easier. Additionally, it support taints, Windows, Linux arm64.

[dduportal]

Karpenter is installed (ref. jenkins-infra/terraform-aws-sponsorship#91) : the cluster is now ready.

Let's continue in #4317