diff --git a/docs/patterns/gitops-multi-cluster-hub-spoke-argocd.md b/docs/patterns/gitops-multi-cluster-hub-spoke-argocd.md new file mode 100644 index 0000000000..0ef45bb13a --- /dev/null +++ b/docs/patterns/gitops-multi-cluster-hub-spoke-argocd.md @@ -0,0 +1,7 @@ +--- +title: GitOps Multi-Cluster Hub-Spoke Topology (ArgoCD) +--- + +{% + include-markdown "../../patterns/gitops/multi-cluster-hub-spoke-argocd/README.md" +%} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/README.md b/patterns/gitops/multi-cluster-hub-spoke-argocd/README.md new file mode 100644 index 0000000000..21a8a35b3e --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/README.md @@ -0,0 +1,159 @@ +# Multi-Cluster centralized hub-spoke topology + +This tutorial guides you through deploying an Amazon EKS cluster with addons configured via ArgoCD in a Multi-Cluster Hub-Spoke topoloy, employing the [GitOps Bridge Pattern](https://github.com/gitops-bridge-dev). + + + + +This example deploys ArgoCD on the Hub cluster (ie. management/control-plane cluster). +The spoke clusters are registered as remote clusters in the Hub Cluster's ArgoCD +The ArgoCD on the Hub Cluster deploy addons and workloads to the spoke clusters + +Each spoke cluster gets deployed an app of apps ArgoCD Application with the name `workloads-${env}` + +## Prerequisites +Before you begin, make sure you have the following command line tools installed: +- git +- terraform +- kubectl +- argocd + +## Fork the Git Repositories + +### Fork the Addon GitOps Repo +1. Fork the git repository for addons [here](https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template). +2. Update the following environment variables to point to your fork by changing the default values: +```shell +export TF_VAR_gitops_addons_org=https://github.com/gitops-bridge-dev +export TF_VAR_gitops_addons_repo=gitops-bridge-argocd-control-plane-template +``` + +## Deploy the Hub EKS Cluster +Change Director to `hub` +```shell +cd hub +``` +Initialize Terraform and deploy the EKS cluster: +```shell +terraform init +terraform apply -auto-approve +``` +Retrieve `kubectl` config, then execute the output command: +```shell +terraform output -raw configure_kubectl +``` + +### Monitor GitOps Progress for Addons +Wait until **all** the ArgoCD applications' `HEALTH STATUS` is `Healthy`. Use Crl+C to exit the `watch` command +```shell +watch kubectl get applications -n argocd +``` + +## Access ArgoCD on Hub Cluster +Access ArgoCD's UI, run the command from the output: +```shell +terraform output -raw access_argocd +``` + +## Verify that ArgoCD Service Accouts has the annotation for IRSA +```shell +kubectl get sa -n argocd argocd-application-controller -o json | jq '.metadata.annotations."eks.amazonaws.com/role-arn"' +kubectl get sa -n argocd argocd-server -o json | jq '.metadata.annotations."eks.amazonaws.com/role-arn"' +``` +The output should match the `arn` for the IAM Role that will assume the IAM Role in spoke/remote clusters +```text +"arn:aws:iam::0123456789:role/hub-spoke-control-plane-argocd-hub" +``` + +## Deploy the Spoke EKS Cluster +Initialize Terraform and deploy the EKS clusters: +```shell +cd ../spokes +./deploy.sh dev +./deploy.sh staging +./deploy.sh prod +``` +Each environment uses a Terraform workspace + +To access Terraform output run the following commands for the particular environment +```shell +terraform workspace select dev +terraform output +``` +```shell +terraform workspace select staging +terraform output +``` +```shell +terraform workspace select prod +terraform output +``` + +Retrieve `kubectl` config, then execute the output command: +```shell +terraform output -raw configure_kubectl +``` + +### Verify ArgoCD Cluster Secret for Spoke has the correct IAM Role to be assume by Hub Cluster +```shell +kubectl get secret -n argocd hub-spoke-dev --template='{{index .data.config | base64decode}}' +``` +Do the same for the other cluster replaced `dev` in `hub-spoke-dev` +The output have a section `awsAuthConfig` with the `clusterName` and the `roleARN` that has write access to the spoke cluster +```json +{ + "tlsClientConfig": { + "insecure": false, + "caData" : "LS0tL...." + }, + "awsAuthConfig" : { + "clusterName": "hub-spoke-dev", + "roleARN": "arn:aws:iam::0123456789:role/hub-spoke-dev-argocd-spoke" + } +} +``` + + +### Verify the Addons on Spoke Clusters +Verify that the addons are ready: +```shell +kubectl get deployment -n kube-system \ + metrics-server +``` + + +### Monitor GitOps Progress for Workloads from Hub Cluster (run on Hub Cluster context) +Watch until **all* the Workloads ArgoCD Applications are `Healthy` +```shell +watch kubectl get -n argocd applications +``` +Wait until the ArgoCD Applications `HEALTH STATUS` is `Healthy`. Crl+C to exit the `watch` command + + +### Verify the Application +Verify that the application configuration is present and the pod is running: +```shell +kubectl get all -n workload +``` + +### Container Metrics +Check the application's CPU and memory metrics: +```shell +kubectl top pods -n workload +``` + +## Destroy the Spoke EKS Clusters +To tear down all the resources and the EKS cluster, run the following command: +```shell +./destroy.sh dev +./destroy.sh staging +./destroy.sh prod +``` + +## Destroy the Hub EKS Clusters +To tear down all the resources and the EKS cluster, run the following command: +Destroy Hub Clusters +```shell +cd ../hub +./destroy.sh +``` diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/addons.yaml b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/addons.yaml new file mode 100644 index 0000000000..89f3602844 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/addons.yaml @@ -0,0 +1,32 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: cluster-addons + namespace: argocd +spec: + syncPolicy: + preserveResourcesOnDeletion: true + generators: + - clusters: + selector: + matchExpressions: + - key: akuity.io/argo-cd-cluster-name + operator: NotIn + values: [in-cluster] + template: + metadata: + name: cluster-addons + spec: + project: default + source: + repoURL: '{{metadata.annotations.addons_repo_url}}' + path: '{{metadata.annotations.addons_repo_basepath}}{{metadata.annotations.addons_repo_path}}' + targetRevision: '{{metadata.annotations.addons_repo_revision}}' + directory: + recurse: true + exclude: exclude/* + destination: + namespace: 'argocd' + name: '{{name}}' + syncPolicy: + automated: {} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/workloads.yaml b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/workloads.yaml new file mode 100644 index 0000000000..c399039367 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/bootstrap/workloads.yaml @@ -0,0 +1,34 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: workloads + namespace: argocd +spec: + syncPolicy: + preserveResourcesOnDeletion: false + generators: + - clusters: + selector: + matchExpressions: + - key: akuity.io/argo-cd-cluster-name + operator: NotIn + values: [in-cluster] + - key: environment + operator: NotIn + values: [control-plane] + template: + metadata: + name: 'workload-{{metadata.labels.environment}}' + spec: + project: default + source: + repoURL: '{{metadata.annotations.workload_repo_url}}' + path: '{{metadata.annotations.workload_repo_basepath}}{{metadata.annotations.workload_repo_path}}' + targetRevision: '{{metadata.annotations.workload_repo_revision}}' + destination: + namespace: 'workload' + name: '{{name}}' + syncPolicy: + automated: {} + syncOptions: + - CreateNamespace=true diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/destroy.sh b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/destroy.sh new file mode 100755 index 0000000000..8ad29aaebb --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/destroy.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +set -uo pipefail + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +ROOTDIR="$(cd ${SCRIPTDIR}/../..; pwd )" +[[ -n "${DEBUG:-}" ]] && set -x + +# Delete the Ingress/SVC before removing the addons +TMPFILE=$(mktemp) +terraform -chdir=$SCRIPTDIR output -raw configure_kubectl > "$TMPFILE" +# check if TMPFILE contains the string "No outputs found" +if [[ ! $(cat $TMPFILE) == *"No outputs found"* ]]; then + source "$TMPFILE" + kubectl delete -n argocd applicationset workloads + kubectl delete -n argocd applicationset cluster-addons + kubectl delete -n argocd applicationset addons-argocd + kubectl delete -n argocd svc argo-cd-argocd-server +fi + +terraform destroy -target="module.gitops_bridge_bootstrap" -auto-approve +terraform destroy -target="module.eks_blueprints_addons" -auto-approve +terraform destroy -target="module.eks" -auto-approve +terraform destroy -target="module.vpc" -auto-approve +terraform destroy -auto-approve diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/main.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/main.tf new file mode 100644 index 0000000000..60710e5dbb --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/main.tf @@ -0,0 +1,297 @@ +provider "aws" { + region = local.region +} +data "aws_caller_identity" "current" {} +data "aws_availability_zones" "available" {} + +provider "helm" { + kubernetes { + host = module.eks.cluster_endpoint + cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + # This requires the awscli to be installed locally where Terraform is executed + args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--region", local.region] + } + } +} + +provider "kubernetes" { + host = module.eks.cluster_endpoint + cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + # This requires the awscli to be installed locally where Terraform is executed + args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--region", local.region] + } +} + +locals { + name = "hub-spoke-${local.environment}" + environment = "control-plane" + region = var.region + + cluster_version = var.kubernetes_version + + vpc_cidr = var.vpc_cidr + azs = slice(data.aws_availability_zones.available.names, 0, 3) + + gitops_addons_url = "${var.gitops_addons_org}/${var.gitops_addons_repo}" + gitops_addons_basepath = var.gitops_addons_basepath + gitops_addons_path = var.gitops_addons_path + gitops_addons_revision = var.gitops_addons_revision + + argocd_namespace = "argocd" + + aws_addons = { + enable_cert_manager = try(var.addons.enable_cert_manager, false) + enable_aws_efs_csi_driver = try(var.addons.enable_aws_efs_csi_driver, false) + enable_aws_fsx_csi_driver = try(var.addons.enable_aws_fsx_csi_driver, false) + enable_aws_cloudwatch_metrics = try(var.addons.enable_aws_cloudwatch_metrics, false) + enable_aws_privateca_issuer = try(var.addons.enable_aws_privateca_issuer, false) + enable_cluster_autoscaler = try(var.addons.enable_cluster_autoscaler, false) + enable_external_dns = try(var.addons.enable_external_dns, false) + enable_external_secrets = try(var.addons.enable_external_secrets, false) + enable_aws_load_balancer_controller = try(var.addons.enable_aws_load_balancer_controller, false) + enable_fargate_fluentbit = try(var.addons.enable_fargate_fluentbit, false) + enable_aws_for_fluentbit = try(var.addons.enable_aws_for_fluentbit, false) + enable_aws_node_termination_handler = try(var.addons.enable_aws_node_termination_handler, false) + enable_karpenter = try(var.addons.enable_karpenter, false) + enable_velero = try(var.addons.enable_velero, false) + enable_aws_gateway_api_controller = try(var.addons.enable_aws_gateway_api_controller, false) + enable_aws_ebs_csi_resources = try(var.addons.enable_aws_ebs_csi_resources, false) + enable_aws_secrets_store_csi_driver_provider = try(var.addons.enable_aws_secrets_store_csi_driver_provider, false) + enable_ack_apigatewayv2 = try(var.addons.enable_ack_apigatewayv2, false) + enable_ack_dynamodb = try(var.addons.enable_ack_dynamodb, false) + enable_ack_s3 = try(var.addons.enable_ack_s3, false) + enable_ack_rds = try(var.addons.enable_ack_rds, false) + enable_ack_prometheusservice = try(var.addons.enable_ack_prometheusservice, false) + enable_ack_emrcontainers = try(var.addons.enable_ack_emrcontainers, false) + enable_ack_sfn = try(var.addons.enable_ack_sfn, false) + enable_ack_eventbridge = try(var.addons.enable_ack_eventbridge, false) + enable_aws_argocd = try(var.addons.enable_aws_argocd, false) + } + oss_addons = { + enable_argocd = try(var.addons.enable_argocd, false) + enable_argo_rollouts = try(var.addons.enable_argo_rollouts, false) + enable_argo_events = try(var.addons.enable_argo_events, false) + enable_argo_workflows = try(var.addons.enable_argo_workflows, false) + enable_cluster_proportional_autoscaler = try(var.addons.enable_cluster_proportional_autoscaler, false) + enable_gatekeeper = try(var.addons.enable_gatekeeper, false) + enable_gpu_operator = try(var.addons.enable_gpu_operator, false) + enable_ingress_nginx = try(var.addons.enable_ingress_nginx, false) + enable_kyverno = try(var.addons.enable_kyverno, false) + enable_kube_prometheus_stack = try(var.addons.enable_kube_prometheus_stack, false) + enable_metrics_server = try(var.addons.enable_metrics_server, false) + enable_prometheus_adapter = try(var.addons.enable_prometheus_adapter, false) + enable_secrets_store_csi_driver = try(var.addons.enable_secrets_store_csi_driver, false) + enable_vpa = try(var.addons.enable_vpa, false) + } + addons = merge( + local.aws_addons, + local.oss_addons, + { kubernetes_version = local.cluster_version }, + { aws_cluster_name = module.eks.cluster_name } + ) + + addons_metadata = merge( + module.eks_blueprints_addons.gitops_metadata, + { + aws_cluster_name = module.eks.cluster_name + aws_region = local.region + aws_account_id = data.aws_caller_identity.current.account_id + aws_vpc_id = module.vpc.vpc_id + }, + { + argocd_iam_role_arn = module.argocd_irsa.iam_role_arn + argocd_namespace = local.argocd_namespace + }, + { + addons_repo_url = local.gitops_addons_url + addons_repo_basepath = local.gitops_addons_basepath + addons_repo_path = local.gitops_addons_path + addons_repo_revision = local.gitops_addons_revision + } + ) + + argocd_apps = { + addons = file("${path.module}/bootstrap/addons.yaml") + workloads = file("${path.module}/bootstrap/workloads.yaml") + } + + tags = { + Blueprint = local.name + GithubRepo = "github.com/gitops-bridge-dev/gitops-bridge" + } +} + +################################################################################ +# GitOps Bridge: Bootstrap +################################################################################ +module "gitops_bridge_bootstrap" { + source = "github.com/gitops-bridge-dev/gitops-bridge-argocd-bootstrap-terraform?ref=v2.0.0" + + cluster = { + cluster_name = module.eks.cluster_name + environment = local.environment + metadata = local.addons_metadata + addons = local.addons + } + apps = local.argocd_apps + argocd = { + namespace = local.argocd_namespace + } +} + +################################################################################ +# ArgoCD EKS Access +################################################################################ +module "argocd_irsa" { + source = "aws-ia/eks-blueprints-addon/aws" + + create_release = false + create_role = true + role_name_use_prefix = false + role_name = "${module.eks.cluster_name}-argocd-hub" + assume_role_condition_test = "StringLike" + create_policy = false + role_policies = { + ArgoCD_EKS_Policy = aws_iam_policy.irsa_policy.arn + } + oidc_providers = { + this = { + provider_arn = module.eks.oidc_provider_arn + namespace = local.argocd_namespace + service_account = "argocd-*" + } + } + tags = local.tags + +} + +resource "aws_iam_policy" "irsa_policy" { + name = "${module.eks.cluster_name}-argocd-irsa" + description = "IAM Policy for ArgoCD Hub" + policy = data.aws_iam_policy_document.irsa_policy.json + tags = local.tags +} + +data "aws_iam_policy_document" "irsa_policy" { + statement { + effect = "Allow" + resources = ["*"] + actions = ["sts:AssumeRole"] + } +} + +################################################################################ +# EKS Blueprints Addons +################################################################################ +module "eks_blueprints_addons" { + source = "aws-ia/eks-blueprints-addons/aws" + version = "~> 1.0" + + cluster_name = module.eks.cluster_name + cluster_endpoint = module.eks.cluster_endpoint + cluster_version = module.eks.cluster_version + oidc_provider_arn = module.eks.oidc_provider_arn + + # Using GitOps Bridge + create_kubernetes_resources = false + + # EKS Blueprints Addons + enable_cert_manager = local.aws_addons.enable_cert_manager + enable_aws_efs_csi_driver = local.aws_addons.enable_aws_efs_csi_driver + enable_aws_fsx_csi_driver = local.aws_addons.enable_aws_fsx_csi_driver + enable_aws_cloudwatch_metrics = local.aws_addons.enable_aws_cloudwatch_metrics + enable_aws_privateca_issuer = local.aws_addons.enable_aws_privateca_issuer + enable_cluster_autoscaler = local.aws_addons.enable_cluster_autoscaler + enable_external_dns = local.aws_addons.enable_external_dns + enable_external_secrets = local.aws_addons.enable_external_secrets + enable_aws_load_balancer_controller = local.aws_addons.enable_aws_load_balancer_controller + enable_fargate_fluentbit = local.aws_addons.enable_fargate_fluentbit + enable_aws_for_fluentbit = local.aws_addons.enable_aws_for_fluentbit + enable_aws_node_termination_handler = local.aws_addons.enable_aws_node_termination_handler + enable_karpenter = local.aws_addons.enable_karpenter + enable_velero = local.aws_addons.enable_velero + enable_aws_gateway_api_controller = local.aws_addons.enable_aws_gateway_api_controller + + tags = local.tags +} + +################################################################################ +# EKS Cluster +################################################################################ +#tfsec:ignore:aws-eks-enable-control-plane-logging +module "eks" { + source = "terraform-aws-modules/eks/aws" + version = "~> 19.13" + + cluster_name = local.name + cluster_version = local.cluster_version + cluster_endpoint_public_access = true + + + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.private_subnets + + eks_managed_node_groups = { + initial = { + instance_types = ["t3.medium"] + + min_size = 3 + max_size = 10 + desired_size = 3 + } + } + # EKS Addons + cluster_addons = { + vpc-cni = { + # Specify the VPC CNI addon should be deployed before compute to ensure + # the addon is configured before data plane compute resources are created + # See README for further details + before_compute = true + most_recent = true # To ensure access to the latest settings provided + configuration_values = jsonencode({ + env = { + # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html + ENABLE_PREFIX_DELEGATION = "true" + WARM_PREFIX_TARGET = "1" + } + }) + } + } + tags = local.tags +} + +################################################################################ +# Supporting Resources +################################################################################ +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "~> 5.0" + + name = local.name + cidr = local.vpc_cidr + + azs = local.azs + private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)] + public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)] + + enable_nat_gateway = true + single_nat_gateway = true + + public_subnet_tags = { + "kubernetes.io/role/elb" = 1 + } + + private_subnet_tags = { + "kubernetes.io/role/internal-elb" = 1 + } + + tags = local.tags +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/outputs.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/outputs.tf new file mode 100644 index 0000000000..c5b089f69a --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/outputs.tf @@ -0,0 +1,57 @@ +output "configure_kubectl" { + description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig" + value = <<-EOT + export KUBECONFIG="/tmp/${module.eks.cluster_name}" + aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name} + EOT +} + + +output "configure_argocd" { + description = "Terminal Setup" + value = <<-EOT + export KUBECONFIG="/tmp/${module.eks.cluster_name}" + aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name} + export ARGOCD_OPTS="--port-forward --port-forward-namespace argocd --grpc-web" + kubectl config set-context --current --namespace argocd + argocd login --port-forward --username admin --password $(argocd admin initial-password | head -1) + echo "ArgoCD Username: admin" + echo "ArgoCD Password: $(kubectl get secrets argocd-initial-admin-secret -n argocd --template="{{index .data.password | base64decode}}")" + echo Port Forward: http://localhost:8080 + kubectl port-forward -n argocd svc/argo-cd-argocd-server 8080:80 + EOT +} + +output "access_argocd" { + description = "ArgoCD Access" + value = <<-EOT + export KUBECONFIG="/tmp/${module.eks.cluster_name}" + aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name} + echo "ArgoCD Username: admin" + echo "ArgoCD Password: $(kubectl get secrets argocd-initial-admin-secret -n argocd --template="{{index .data.password | base64decode}}")" + echo "ArgoCD URL: https://$(kubectl get svc -n argocd argo-cd-argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')" + EOT +} + + +output "argocd_iam_role_arn" { + description = "IAM Role for ArgoCD Cluster Hub, use to connect to spoke clusters" + value = module.argocd_irsa.iam_role_arn +} + +output "cluster_name" { + description = "Cluster Hub name" + value = module.eks.cluster_name +} +output "cluster_endpoint" { + description = "Cluster Hub endpoint" + value = module.eks.cluster_endpoint +} +output "cluster_certificate_authority_data" { + description = "Cluster Hub certificate_authority_data" + value = module.eks.cluster_certificate_authority_data +} +output "cluster_region" { + description = "Cluster Hub region" + value = local.region +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/variables.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/variables.tf new file mode 100644 index 0000000000..2e2c8462ee --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/variables.tf @@ -0,0 +1,53 @@ +variable "vpc_cidr" { + description = "VPC CIDR" + type = string + default = "10.0.0.0/16" +} +variable "region" { + description = "AWS region" + type = string + default = "us-west-2" +} +variable "kubernetes_version" { + description = "Kubernetes version" + type = string + default = "1.28" +} +variable "addons" { + description = "Kubernetes addons" + type = any + default = { + enable_aws_load_balancer_controller = true + enable_metrics_server = true + # Enable argocd with IRSA + enable_aws_argocd = true + # Disable argocd without IRSA + enable_argocd = false + } +} +# Addons Git +variable "gitops_addons_org" { + description = "Git repository org/user contains for addons" + type = string + default = "https://github.com/gitops-bridge-dev" +} +variable "gitops_addons_repo" { + description = "Git repository contains for addons" + type = string + default = "gitops-bridge-argocd-control-plane-template" +} +variable "gitops_addons_revision" { + description = "Git repository revision/branch/ref for addons" + type = string + default = "main" +} +variable "gitops_addons_basepath" { + description = "Git repository base path for addons" + type = string + default = "" +} +variable "gitops_addons_path" { + description = "Git repository path for addons" + type = string + default = "bootstrap/control-plane/addons" +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/versions.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/versions.tf new file mode 100644 index 0000000000..2de60d58ee --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/hub/versions.tf @@ -0,0 +1,25 @@ +terraform { + required_version = ">= 1.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.67.0" + } + helm = { + source = "hashicorp/helm" + version = ">= 2.10.1" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "2.22.0" + } + } + + # ## Used for end-to-end testing on project; update to suit your needs + # backend "s3" { + # bucket = "terraform-ssp-github-actions-state" + # region = "us-west-2" + # key = "e2e/ipv4-prefix-delegation/terraform.tfstate" + # } +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/deploy.sh b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/deploy.sh new file mode 100755 index 0000000000..0dad2a7030 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/deploy.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +if [[ $# -eq 0 ]] ; then + echo "No arguments supplied" + echo "Usage: deploy.sh " + echo "Example: deploy.sh dev" + exit 1 +fi +env=$1 +echo "Deploying $env with "workspaces/${env}.tfvars" ..." + +set -x + +terraform workspace new $env +terraform workspace select $env +terraform init +terraform apply -var-file="workspaces/${env}.tfvars" diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/destroy.sh b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/destroy.sh new file mode 100755 index 0000000000..cc2a333fdd --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/destroy.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +set -uo pipefail + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +ROOTDIR="$(cd ${SCRIPTDIR}/../..; pwd )" +[[ -n "${DEBUG:-}" ]] && set -x + + +if [[ $# -eq 0 ]] ; then + echo "No arguments supplied" + echo "Usage: destroy.sh " + echo "Example: destroy.sh dev" + exit 1 +fi +env=$1 +echo "Destroying $env ..." +terraform workspace select $env + +terraform destroy -auto-approve -var-file="workspaces/${env}.tfvars" -target="module.gitops_bridge_bootstrap" -auto-approve +terraform destroy -auto-approve -var-file="workspaces/${env}.tfvars" -target="module.eks_blueprints_addons" -auto-approve +terraform destroy -auto-approve -var-file="workspaces/${env}.tfvars" -target="module.eks" -auto-approve +terraform destroy -auto-approve -var-file="workspaces/${env}.tfvars" -target="module.vpc" -auto-approve +terraform destroy -auto-approve -var-file="workspaces/${env}.tfvars" diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/main.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/main.tf new file mode 100644 index 0000000000..28b328e5a3 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/main.tf @@ -0,0 +1,323 @@ +provider "aws" { + region = local.region +} +data "aws_caller_identity" "current" {} +data "aws_availability_zones" "available" {} + + +data "terraform_remote_state" "cluster_hub" { + backend = "local" + + config = { + path = "${path.module}/../hub/terraform.tfstate" + } +} + +################################################################################ +# Kubernetes Access for Hub Cluster +################################################################################ + +provider "kubernetes" { + host = data.terraform_remote_state.cluster_hub.outputs.cluster_endpoint + cluster_ca_certificate = base64decode(data.terraform_remote_state.cluster_hub.outputs.cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + # This requires the awscli to be installed locally where Terraform is executed + args = ["eks", "get-token", "--cluster-name", data.terraform_remote_state.cluster_hub.outputs.cluster_name, "--region", data.terraform_remote_state.cluster_hub.outputs.cluster_region] + } + alias = "hub" +} + +################################################################################ +# Kubernetes Access for Spoke Cluster +################################################################################ + +provider "kubernetes" { + host = module.eks.cluster_endpoint + cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + # This requires the awscli to be installed locally where Terraform is executed + args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name, "--region", local.region] + } +} + + + +locals { + name = "hub-spoke-${terraform.workspace}" + environment = terraform.workspace + region = var.region + + cluster_version = var.kubernetes_version + + vpc_cidr = var.vpc_cidr + azs = slice(data.aws_availability_zones.available.names, 0, 3) + + gitops_addons_url = "${var.gitops_addons_org}/${var.gitops_addons_repo}" + gitops_addons_basepath = var.gitops_addons_basepath + gitops_addons_path = var.gitops_addons_path + gitops_addons_revision = var.gitops_addons_revision + + gitops_workload_org = var.gitops_workload_org + gitops_workload_repo = var.gitops_workload_repo + gitops_workload_basepath = var.gitops_workload_basepath + gitops_workload_path = var.gitops_workload_path + gitops_workload_revision = var.gitops_workload_revision + gitops_workload_url = "${local.gitops_workload_org}/${local.gitops_workload_repo}" + + aws_addons = { + enable_cert_manager = try(var.addons.enable_cert_manager, false) + enable_aws_efs_csi_driver = try(var.addons.enable_aws_efs_csi_driver, false) + enable_aws_fsx_csi_driver = try(var.addons.enable_aws_fsx_csi_driver, false) + enable_aws_cloudwatch_metrics = try(var.addons.enable_aws_cloudwatch_metrics, false) + enable_aws_privateca_issuer = try(var.addons.enable_aws_privateca_issuer, false) + enable_cluster_autoscaler = try(var.addons.enable_cluster_autoscaler, false) + enable_external_dns = try(var.addons.enable_external_dns, false) + enable_external_secrets = try(var.addons.enable_external_secrets, false) + enable_aws_load_balancer_controller = try(var.addons.enable_aws_load_balancer_controller, false) + enable_fargate_fluentbit = try(var.addons.enable_fargate_fluentbit, false) + enable_aws_for_fluentbit = try(var.addons.enable_aws_for_fluentbit, false) + enable_aws_node_termination_handler = try(var.addons.enable_aws_node_termination_handler, false) + enable_karpenter = try(var.addons.enable_karpenter, false) + enable_velero = try(var.addons.enable_velero, false) + enable_aws_gateway_api_controller = try(var.addons.enable_aws_gateway_api_controller, false) + enable_aws_ebs_csi_resources = try(var.addons.enable_aws_ebs_csi_resources, false) + enable_aws_secrets_store_csi_driver_provider = try(var.addons.enable_aws_secrets_store_csi_driver_provider, false) + enable_ack_apigatewayv2 = try(var.addons.enable_ack_apigatewayv2, false) + enable_ack_dynamodb = try(var.addons.enable_ack_dynamodb, false) + enable_ack_s3 = try(var.addons.enable_ack_s3, false) + enable_ack_rds = try(var.addons.enable_ack_rds, false) + enable_ack_prometheusservice = try(var.addons.enable_ack_prometheusservice, false) + enable_ack_emrcontainers = try(var.addons.enable_ack_emrcontainers, false) + enable_ack_sfn = try(var.addons.enable_ack_sfn, false) + enable_ack_eventbridge = try(var.addons.enable_ack_eventbridge, false) + enable_aws_argocd = try(var.addons.enable_aws_argocd, false) + } + oss_addons = { + enable_argocd = try(var.addons.enable_argocd, false) + enable_argo_rollouts = try(var.addons.enable_argo_rollouts, false) + enable_argo_events = try(var.addons.enable_argo_events, false) + enable_argo_workflows = try(var.addons.enable_argo_workflows, false) + enable_cluster_proportional_autoscaler = try(var.addons.enable_cluster_proportional_autoscaler, false) + enable_gatekeeper = try(var.addons.enable_gatekeeper, false) + enable_gpu_operator = try(var.addons.enable_gpu_operator, false) + enable_ingress_nginx = try(var.addons.enable_ingress_nginx, false) + enable_kyverno = try(var.addons.enable_kyverno, false) + enable_kube_prometheus_stack = try(var.addons.enable_kube_prometheus_stack, false) + enable_metrics_server = try(var.addons.enable_metrics_server, false) + enable_prometheus_adapter = try(var.addons.enable_prometheus_adapter, false) + enable_secrets_store_csi_driver = try(var.addons.enable_secrets_store_csi_driver, false) + enable_vpa = try(var.addons.enable_vpa, false) + } + addons = merge( + local.aws_addons, + local.oss_addons, + { kubernetes_version = local.cluster_version }, + { aws_cluster_name = module.eks.cluster_name } + ) + + addons_metadata = merge( + module.eks_blueprints_addons.gitops_metadata, + { + aws_cluster_name = module.eks.cluster_name + aws_region = local.region + aws_account_id = data.aws_caller_identity.current.account_id + aws_vpc_id = module.vpc.vpc_id + }, + { + addons_repo_url = local.gitops_addons_url + addons_repo_basepath = local.gitops_addons_basepath + addons_repo_path = local.gitops_addons_path + addons_repo_revision = local.gitops_addons_revision + }, + { + workload_repo_url = local.gitops_workload_url + workload_repo_basepath = local.gitops_workload_basepath + workload_repo_path = local.gitops_workload_path + workload_repo_revision = local.gitops_workload_revision + } + ) + + tags = { + Blueprint = local.name + GithubRepo = "github.com/gitops-bridge-dev/gitops-bridge" + } +} + +################################################################################ +# GitOps Bridge: Bootstrap for Hub Cluster +################################################################################ +module "gitops_bridge_bootstrap_hub" { + source = "github.com/gitops-bridge-dev/gitops-bridge-argocd-bootstrap-terraform?ref=v2.0.0" + + # The ArgoCD remote cluster secret is deploy on hub cluster not on spoke clusters + providers = { + kubernetes = kubernetes.hub + } + + install = false # We are not installing argocd via helm on hub cluster + cluster = { + cluster_name = module.eks.cluster_name + environment = local.environment + metadata = local.addons_metadata + addons = local.addons + server = module.eks.cluster_endpoint + config = <<-EOT + { + "tlsClientConfig": { + "insecure": false, + "caData" : "${module.eks.cluster_certificate_authority_data}" + }, + "awsAuthConfig" : { + "clusterName": "${module.eks.cluster_name}", + "roleARN": "${aws_iam_role.spoke.arn}" + } + } + EOT + } +} + +################################################################################ +# ArgoCD EKS Access +################################################################################ +resource "aws_iam_role" "spoke" { + name = "${module.eks.cluster_name}-argocd-spoke" + assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json +} + +data "aws_iam_policy_document" "assume_role_policy" { + statement { + actions = ["sts:AssumeRole"] + principals { + type = "AWS" + identifiers = [data.terraform_remote_state.cluster_hub.outputs.argocd_iam_role_arn] + } + } +} + + + +################################################################################ +# EKS Blueprints Addons +################################################################################ +module "eks_blueprints_addons" { + source = "aws-ia/eks-blueprints-addons/aws" + version = "~> 1.0" + + cluster_name = module.eks.cluster_name + cluster_endpoint = module.eks.cluster_endpoint + cluster_version = module.eks.cluster_version + oidc_provider_arn = module.eks.oidc_provider_arn + + # Using GitOps Bridge + create_kubernetes_resources = false + + # EKS Blueprints Addons + enable_cert_manager = local.aws_addons.enable_cert_manager + enable_aws_efs_csi_driver = local.aws_addons.enable_aws_efs_csi_driver + enable_aws_fsx_csi_driver = local.aws_addons.enable_aws_fsx_csi_driver + enable_aws_cloudwatch_metrics = local.aws_addons.enable_aws_cloudwatch_metrics + enable_aws_privateca_issuer = local.aws_addons.enable_aws_privateca_issuer + enable_cluster_autoscaler = local.aws_addons.enable_cluster_autoscaler + enable_external_dns = local.aws_addons.enable_external_dns + enable_external_secrets = local.aws_addons.enable_external_secrets + enable_aws_load_balancer_controller = local.aws_addons.enable_aws_load_balancer_controller + enable_fargate_fluentbit = local.aws_addons.enable_fargate_fluentbit + enable_aws_for_fluentbit = local.aws_addons.enable_aws_for_fluentbit + enable_aws_node_termination_handler = local.aws_addons.enable_aws_node_termination_handler + enable_karpenter = local.aws_addons.enable_karpenter + enable_velero = local.aws_addons.enable_velero + enable_aws_gateway_api_controller = local.aws_addons.enable_aws_gateway_api_controller + + tags = local.tags +} + +################################################################################ +# EKS Cluster +################################################################################ +#tfsec:ignore:aws-eks-enable-control-plane-logging +module "eks" { + source = "terraform-aws-modules/eks/aws" + version = "~> 19.13" + + cluster_name = local.name + cluster_version = local.cluster_version + cluster_endpoint_public_access = true + + + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.private_subnets + + manage_aws_auth_configmap = true + aws_auth_roles = [ + # Granting access to ArgoCD from hub cluster + { + rolearn = aws_iam_role.spoke.arn + username = "gitops-role" + groups = [ + "system:masters" + ] + }, + ] + + eks_managed_node_groups = { + initial = { + instance_types = ["t3.medium"] + + min_size = 3 + max_size = 10 + desired_size = 3 + } + } + # EKS Addons + cluster_addons = { + vpc-cni = { + # Specify the VPC CNI addon should be deployed before compute to ensure + # the addon is configured before data plane compute resources are created + # See README for further details + before_compute = true + most_recent = true # To ensure access to the latest settings provided + configuration_values = jsonencode({ + env = { + # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html + ENABLE_PREFIX_DELEGATION = "true" + WARM_PREFIX_TARGET = "1" + } + }) + } + } + tags = local.tags +} + +################################################################################ +# Supporting Resources +################################################################################ +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "~> 5.0" + + name = local.name + cidr = local.vpc_cidr + + azs = local.azs + private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)] + public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)] + + enable_nat_gateway = true + single_nat_gateway = true + + public_subnet_tags = { + "kubernetes.io/role/elb" = 1 + } + + private_subnet_tags = { + "kubernetes.io/role/internal-elb" = 1 + } + + tags = local.tags +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/outputs.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/outputs.tf new file mode 100644 index 0000000000..e398229ac0 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/outputs.tf @@ -0,0 +1,7 @@ +output "configure_kubectl" { + description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig" + value = <<-EOT + export KUBECONFIG="/tmp/${module.eks.cluster_name}" + aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name} + EOT +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/variables.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/variables.tf new file mode 100644 index 0000000000..570325119c --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/variables.tf @@ -0,0 +1,69 @@ +variable "region" { + description = "AWS region" + type = string +} +variable "vpc_cidr" { + description = "VPC CIDR" + type = string +} +variable "kubernetes_version" { + description = "EKS version" + type = string +} +variable "addons" { + description = "Kubernetes addons" + type = any +} +# Addons Git +variable "gitops_addons_org" { + description = "Git repository org/user contains for addons" + type = string + default = "https://github.com/gitops-bridge-dev" +} +variable "gitops_addons_repo" { + description = "Git repository contains for addons" + type = string + default = "gitops-bridge-argocd-control-plane-template" +} +variable "gitops_addons_revision" { + description = "Git repository revision/branch/ref for addons" + type = string + default = "main" +} +variable "gitops_addons_basepath" { + description = "Git repository base path for addons" + type = string + default = "" +} +variable "gitops_addons_path" { + description = "Git repository path for addons" + type = string + default = "bootstrap/control-plane/addons" +} + +# Workloads Git +variable "gitops_workload_org" { + description = "Git repository org/user contains for workload" + type = string + default = "https://github.com/argoproj" +} +variable "gitops_workload_repo" { + description = "Git repository contains for workload" + type = string + default = "argocd-example-apps" +} +variable "gitops_workload_revision" { + description = "Git repository revision/branch/ref for workload" + type = string + default = "master" +} +variable "gitops_workload_basepath" { + description = "Git repository base path for workload" + type = string + default = "" +} +variable "gitops_workload_path" { + description = "Git repository path for workload" + type = string + default = "helm-guestbook" +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/versions.tf b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/versions.tf new file mode 100644 index 0000000000..6ff7c991ec --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/spokes/versions.tf @@ -0,0 +1,21 @@ +terraform { + required_version = ">= 1.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.67.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "2.22.0" + } + } + + # ## Used for end-to-end testing on project; update to suit your needs + # backend "s3" { + # bucket = "terraform-ssp-github-actions-state" + # region = "us-west-2" + # key = "e2e/ipv4-prefix-delegation/terraform.tfstate" + # } +} diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio b/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio new file mode 100644 index 0000000000..083d637330 --- /dev/null +++ b/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio @@ -0,0 +1 @@ +7Vvbcps6FP0aPyaDwPjyGGPnnE7bmc7JQ+NHxcigRkZUyDHu1x8JhEESvSWxIRMSZ4yWLoi1tLe2duyRF+zyfxhM4880RGTkOmE+8pYj13Wd8Vi8SeRYIgDM/BKJGA4VVgN3+AdSoKPQPQ5RpjXklBKOUx3c0CRBG65hkDF60JttKdHvmsIIWcDdBhIb/YpDHpfozHdq/F+Eo7i6M3BUzQ5WjRWQxTCkhwbkrUZewCjl5dUuDxCR7FW8lP1uf1J7mhhDCf+TDph9/b7+L/h8e3+TTO7Xa/QD7K68cpQnSPbqgdVk+bFigNF9EiI5iDPyFocYc3SXwo2sPQjRBRbzHRElIC7VcIhxlP90nuD09GLdILpDnB1FE9VhqvhSK2ZcrYRDTT+YKSxuUj9WIFSSR6eha1bEhSLmL0ga/56k7BHxTaw4SilOeDEJfyFeYlpB+eeLpoFErl2/BWzDpjYI7GbiDbTdwQTbsKkNAruZLFWz1sE2bOrbMzZ7g5bewOgtXt6C7jnBCQpOFi453tKEB5RQVvDvid9bqegiYjDESKu7nc5WzrhRt8RMDIRpIuoTyuSqWmwxIY0+S8cPwFTgGWf0ETVqtsWPqAlhFp/MQq54LLzGJ/iAyBeaYTX8A+Wc7hoNbgiOZAWn0nKgKm3ErBDTTUk+ofKHwK3KasXJW8IsLenY4lzOYyEcTCord3kknfE1PGTja4Yyumcb9GEj57MQxfJKb4Ues4sZr99iu/65TNe3TBeGIU2yzr0ccA2mpjZT8xamvHMxNbGYOlD2SCgM+0eW37KsLkrWtGVHmBCurFQja/J9T6uKq6yw5xvRwAVpXpBT1YurSL6vPt5VY4mplcOVNZYMglCuc617q4QmyHBtCrL8jumedjgM5W1axdXlfw19PcMYXFtf75L6VgM3yEahiAtVUe4ZNKIJJKsaNWip23yi0tcX+nxDnB+VU4d7TnX1BFvseC/7i21RFddquKKwzLXSsbEXqkHdYpwc88YworSu7iCu60Fk4aipKB/y1xoKTor95Bfkzct2HLII8d/FnvaaYIhAjp/0eby+wuAtK9yBWmDcqVx2yP2G5HojBgnmL5RYdf0izz+1d/fmunf3pobbLh9A9TIWymkaz18785bNuuOAxnN0SlzH3vAufMitQqrhlDuccodT7svN96LHXNCWxxsOJM8XeGII3HLgvOiBBPxBDrLrTawHmVpg53uGTWzYxIZN7Jnme9lNzE5B9iRZ6/kGVV0na4GdgOxPttZkq/NsLZgN0dE5o6PO07Wgh0d838xpdx8dVTIN0dEQHQ3R0cvN96LRUfUI/YuO/FnPoqPKrfUyOjLZ6jw6cofc0atGR/68Z9GRa+eO4v1DqZxwoYJlcpUSKLjspyblPatPX552EOvfoa96GgYtTqw1XDufbHYuK0sFEaVwIXp6z3IZnzPzJ52rZZ/IG2plHEY4id6zYqaB9UAyOy3QkCxlNHzPepkBcA/0sk/5NyyiwbLzmM7MeLnni4BFsf7KQPkJifqbF97qfw== \ No newline at end of file diff --git a/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio.png b/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio.png new file mode 100644 index 0000000000..7ac6de28b8 Binary files /dev/null and b/patterns/gitops/multi-cluster-hub-spoke-argocd/static/gitops-bridge-multi-cluster-hup-spoke.drawio.png differ