k8splayground-core-infra

Handles terraform for core infra of Kubernetes playground: Networking, clusters, etc.

Primarily driven by EKS Blueprints:

• repo
• docs

However, given the change in direction for v5 of blueprints, I will not be adopting the actual root blueprint module. Instead, I will be consuming the examples as references.

Architecture and properties:

• Karpenter running on Fargate, managing the provisioning of node groups. Running in fargate protects against the failure mode of "no nodes, so can't run the controller that adds nodes".
• coredns, kube-proxy, vpc-cni, as required networking plugins, configured to support karpenter's initialization.
• fargate profiles for kube-system and karpenter .
• ArgoCD, for deploying everything else.

Cluster Add-on Management:

Given the changes to eks blueprints, managing add-ons is in a bit of a weird state. In general, the changes appears to be directed at:

• creating a new reference repo
• changing the organization and style of add-on modules: instead of each add-on requiring a full module in the repo, most add-ons will be converted to use a generalized eks-blueprints-addon module, that handles provisioning of IRSA and (potentially) a helm-chart.
  • note: the link above is to a temporary copy of the module, while the long-term repo and registry entry are vetted for release. Long term links (that should eventually become live) are below:
  • terraform module registry
  • github repo

Because the repos are not ready, are in pretty constant flux and are buggy in fundamental ways, I'm taking the following approach (as of this writing. Likely will change. I'll try to keep this section updated):

• When consuming a module sourced from any of these systems, will pin a version/commit, not a branch.
• Will try to keep up-to-date with the modules, and simplify the below when possible.
• Avoid using the root module at terraform-aws-eks-blueprints-addons, but will instead use it as a reference. (Might change if my bug gets resolved?)
• Continue using the blueprint ArgoCD module: it's well-abstracted, and gives the needed toggles.
• Will use the eks-blueprints-addon module for bootstrapping IRSA: it's well-abstracted, and designed to be used with the argocd module.
  • Will only use it to manage IRSA, and pass values into argoCD. Will not use it to render the helm release, because that's argo's job.
• Each add-on will get a top-level <addonName>.tf file here, that will contain its policy document and use of the add-on module.

A note on karpenter:

Our basic goal is to have as much of our system git-ops'd as possible. However, karpenter-in-fargate is the only system that actual creates nodes, so it needs to be fully ready before argoCD comes online. Therefor, we directly deploy karpenter's helmchart and configuration outside of the normal scope of argocd.