🏗 Build Analytical Platform Compute EKS

[jacobwoffenden]

User Story

As an Analytical Platform Product Engineer
I need to host applications that can't live on Cloud Platform
So that we can run the service

Value / Purpose

DPAT's EKS lives in DPAT account, and can't easily be renamed, my proposal it to provision a new Modernisation Platform (analytical-platform-compute), and rebuild our EKS cluster (without any Data Platform components)

Useful Contacts

@jacobwoffenden

User Types

Analytical Platform Product Engineering

Hypothesis

If we rebuild our new EKS cluster in Analytical Platform Compute
Then we have home for Analytical Platform Dashboard, Analytical Platform Tools, and Analytical Platform Airflow

Proposal

• Request new Modernisation Platform accounts (analytical-platform-compute)
• Retire new Modernisation Platform accounts (data-platform-compute)
• Build out EKS cluster

Additional Information

DPAT EKS currently hosts:

• Data Platform Assets (web service serving images)
• Data Platform's Open Metadata and DataHub instances
• GitHub Actions Self Hosted Runner capability

Definition of Done

• Request new Modernisation Platform accounts (analytical-platform-compute)
• Retire new Modernisation Platform accounts (data-platform-compute)
  • 🔥 Decommission data-platform-compute modernisation-platform#7071
• Build out EKS cluster

[jacobwoffenden]

02/05/24 update:

• Build out of cluster in AP Dev continues in 🧑‍🔬 Analytical Platform Compute testing #4259
  • Switched to EKS API for authentication management
  • Enabled CloudWatch Observability addon
• VPCs have been been rearchitected to consume Airflow's VPC range with the intention that our networks will replaces theirs ministryofjustice/modernisation-platform-environments@6577715

[jacobwoffenden]

Cluster creation blocked as MemberInfrastructureAccess cannot create OIDC provider due to org SCP

Error: creating IAM OIDC Provider: operation error IAM: CreateOpenIDConnectProvider, https response error StatusCode: 403, RequestID: 56134a29-6a14-430b-879a-2f4fa59f0e41, api error AccessDenied: User: arn:aws:sts::381491960855:assumed-role/MemberInfrastructureAccess/aws-go-sdk-1715608907993994861 is not authorized to perform: iam:CreateOpenIDConnectProvider on resource: arn:aws:iam::381491960855:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/1972AFFBD0701A0D1FD291E34F7D1287 with an explicit deny in a service control policy

[jacobwoffenden]

13/05/24 update:

• In an effort to not build the cluster bit-by-bit, I configured VPC CNI to use IRSA immediately, but theres a dependency on the cluster being "ready" so that the IRSA enabled role can create... 🐔 & 🥚

  • I switched back to the default of VPC CNI using the node's role, and enabling the VPC CNI policy, but I will perform another experiment tomorrow of the CNI policy being attached to the node and VPC CNI using IRSA immediately (in Analytical Platform Development)

• However cluster creation was ultimately blocked by an organisational SCP denying iam: CreateOpenIDConnectProvider

  • @dms1981 added the permissions to MemberInfrastructureAccess which exposed the SCP
  • @ewastempel removed the accounts from the OU, and introduced another error because GitHub Action couldn't assume the backend role

[jacobwoffenden]

15/05/24 update:

• Work was carried out with @ministryofjustice/modernisation-platform to unblock EKS deployment
  • 🛂 Add aws_iam_openid_connect_provider to plan evaluator modernisation-platform-environments#6071
  • Removing OIDC actions from the MP SCP aws-root-account#906
  • 🔥 Remove OpenIDConnectProvider actions from deny modernisation-platform#6998
• Cluster has been deployed, but no extra components deployed
• Work halted to allow 🔧 QuickSight Terraform #3881

[jacobwoffenden]

16/05/24 update:

• Work halted still, but agreed with @julialawrence that I'd deploy the following charts via Helm instead of using the native Amazon CloudWatch Observability EKS add-on, because I can't see how we'd pass in arguments to use CMK or IRSA/Pod Identity
  • https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit
  • https://github.com/aws/eks-charts/tree/master/stable/aws-cloudwatch-metrics

[jacobwoffenden]

20/05/24 update:

• aws-cloudwatch-metrics installed but is configured to use hostNetwork: true as it currently doesn't support either IRSA or EKS Pod Identity 😭
• aws-for-fluent-bit installed, publishing to CMK encrypted CloudWatch log group using IRSA 🎉

[jacobwoffenden]

21/05/24 update:

• Added support for reading from remote AMP workspaces
  • ✨ Allow reading of remote Amazon Managed Prometheus workspaces observability-platform#86
    • Pending working GitHub Actions 🙏 ✨ Add support for reading remote Amazon Managed Prometheus workspaces modernisation-platform-environments#6184
• Added Kyverno

[jacobwoffenden]

22/05/24 update:

• No work carried out

[jacobwoffenden]

Moving to blocked due to aws/containers-roadmap#2359, cannot proceed as plans aren't successful. Have raised in #ext-slack

[jacobwoffenden]

1.30 compatibility has been added to aws-guardduty-agent

[jacobwoffenden]

Span out into:

• 🚚 Migrate GitHub Actions Runners from DPAT to APC #4422
• 📖 Switch APC to kube-prometheus-stack #4424
• 🚢 Import APC's AMP into AMP module #4425