From 20cb24083ff16e39022f5ab0ab0b2390311a823c Mon Sep 17 00:00:00 2001 From: Ellis Tarn Date: Fri, 23 Jul 2021 14:31:17 -0700 Subject: [PATCH] Revisions to the Getting Started guide --- pkg/cloudprovider/aws/docs/README.md | 169 ------------------ pkg/cloudprovider/aws/docs/eks-config.yaml | 43 ----- .../aws/docs/examples/provisioner.yaml | 31 ---- .../content/en/docs/getting-started/_index.md | 107 +++++------ .../docs/getting-started/cloudformation.yaml | 0 .../{eks-config.yaml => eksctl.yaml} | 1 - 6 files changed, 41 insertions(+), 310 deletions(-) delete mode 100644 pkg/cloudprovider/aws/docs/README.md delete mode 100644 pkg/cloudprovider/aws/docs/eks-config.yaml delete mode 100644 pkg/cloudprovider/aws/docs/examples/provisioner.yaml rename pkg/cloudprovider/aws/docs/karpenter.cloudformation.yaml => website/content/en/docs/getting-started/cloudformation.yaml (100%) rename website/content/en/docs/getting-started/{eks-config.yaml => eksctl.yaml} (98%) diff --git a/pkg/cloudprovider/aws/docs/README.md b/pkg/cloudprovider/aws/docs/README.md deleted file mode 100644 index db8f51797848..000000000000 --- a/pkg/cloudprovider/aws/docs/README.md +++ /dev/null @@ -1,169 +0,0 @@ -# AWS - -This guide will provide a complete Karpenter installation for AWS. -These steps are opinionated and may need to be adapted for your use case. - -> This guide should take less than 1 hour to complete and cost less than $.25 - -## Environment -```bash -export CLUSTER_NAME=$USER-karpenter-demo -export AWS_DEFAULT_REGION=us-west-2 -AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) -KARPENTER_VERSION=$(curl -fsSL \ - https://api.github.com/repos/awslabs/karpenter/releases/latest \ - | jq -r '.tag_name') -``` - -### Create a Cluster - -Karpenter can run anywhere, including on self-managed node groups, [managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html), or [AWS Fargate](https://aws.amazon.com/fargate/). -This demo will run Karpenter on Fargate, which means all EC2 instances added to this cluster will be controlled by Karpenter. - -```bash -curl -fsSL https://raw.githubusercontent.com/awslabs/karpenter/"${KARPENTER_VERSION}"/pkg/cloudprovider/aws/docs/eks-config.yaml \ - | envsubst \ - | eksctl create cluster -f - -``` - -Tag the cluster subnets with the required tags for Karpenter auto discovery. - -> If you are using a cluster with version 1.18 or below you can skip this step. -More [detailed here](https://github.com/awslabs/karpenter/issues/404#issuecomment-845283904). - -```bash -SUBNET_IDS=$(aws cloudformation describe-stacks \ - --stack-name eksctl-${CLUSTER_NAME}-cluster \ - --query 'Stacks[].Outputs[?OutputKey==`SubnetsPrivate`].OutputValue' \ - --output text) - -aws ec2 create-tags \ - --resources $(echo ${SUBNET_IDS//,/ }) \ - --tags Key="kubernetes.io/cluster/${CLUSTER_NAME}",Value= -``` - -### Setup IRSA, Karpenter Controller Role, and Karpenter Node Role -We recommend using [CloudFormation](https://aws.amazon.com/cloudformation/) and [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA) to manage these permissions. -For production use, please review and restrict these permissions for your use case. - -> For IRSA to work your cluster needs an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) - -```bash -# Creates IAM resources used by Karpenter -TEMPOUT=$(mktemp) -curl -fsSL https://raw.githubusercontent.com/awslabs/karpenter/"${KARPENTER_VERSION}"/pkg/cloudprovider/aws/docs/karpenter.cloudformation.yaml > $TEMPOUT \ -&& aws cloudformation deploy \ - --stack-name Karpenter-${CLUSTER_NAME} \ - --template-file ${TEMPOUT} \ - --capabilities CAPABILITY_NAMED_IAM \ - --parameter-overrides ClusterName=${CLUSTER_NAME} - -# Add the karpenter node role to your aws-auth configmap, allowing nodes with this role to connect to the cluster. -eksctl create iamidentitymapping \ - --username system:node:{{EC2PrivateDNSName}} \ - --cluster ${CLUSTER_NAME} \ - --arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} \ - --group system:bootstrappers \ - --group system:nodes -``` - -### Install Karpenter - -Use [`helm`](https://helm.sh/) to deploy Karpenter to the cluster. -For additional values, see [the helm chart values](https://github.com/awslabs/karpenter/blob/main/charts/karpenter/values.yaml) - -> We created a Kubernetes service account with our cluster so we don't need the helm chart to do that. - -```bash -helm repo add karpenter https://awslabs.github.io/karpenter/charts -helm repo update -helm upgrade --install karpenter karpenter/karpenter \ - --namespace karpenter --set serviceAccount.create=false -``` - -### (Optional) Enable Verbose Logging -```bash -kubectl patch deployment karpenter-controller \ - -n karpenter --type='json' \ - -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["--verbose"]}]' -``` - -### Create a Provisioner -Create a default Provisioner that launches nodes configured with cluster name and endpoint. -```bash -cat < Karpenter automatically adds a node finalizer to properly cordon and drain nodes before they are terminated. -```bash -kubectl delete node $NODE_NAME -``` - -### Cleanup -> To avoid additional costs make sure you delete all ec2 instances before deleting the other cluster resources. -```bash -helm delete karpenter -n karpenter -aws cloudformation delete-stack --stack-name Karpenter-${CLUSTER_NAME} -aws ec2 describe-launch-templates \ - | jq -r ".LaunchTemplates[].LaunchTemplateName" \ - | grep -i Karpenter-${CLUSTER_NAME} \ - | xargs -I{} aws ec2 delete-launch-template --launch-template-name {} -``` - -If you created a cluster during this process you also will need to delete the cluster. -```bash -eksctl delete cluster --name ${CLUSTER_NAME} -``` diff --git a/pkg/cloudprovider/aws/docs/eks-config.yaml b/pkg/cloudprovider/aws/docs/eks-config.yaml deleted file mode 100644 index fba429b22b2a..000000000000 --- a/pkg/cloudprovider/aws/docs/eks-config.yaml +++ /dev/null @@ -1,43 +0,0 @@ ---- -apiVersion: eksctl.io/v1alpha5 -kind: ClusterConfig -metadata: - name: ${CLUSTER_NAME} - region: ${AWS_DEFAULT_REGION} - -iam: - withOIDC: true - serviceAccounts: - - metadata: - name: karpenter - namespace: karpenter - attachPolicy: - Version: "2012-10-17" - Statement: - - Effect: Allow - Resource: "*" - Action: - # Write Operations - - "ec2:CreateLaunchTemplate" - - "ec2:CreateFleet" - - "ec2:RunInstances" - - "ec2:CreateTags" - - "iam:PassRole" - - "ec2:TerminateInstances" - # Read Operations - - "ec2:DescribeLaunchTemplates" - - "ec2:DescribeInstances" - - "ec2:DescribeSecurityGroups" - - "ec2:DescribeSubnets" - - "ec2:DescribeInstanceTypes" - - "ec2:DescribeInstanceTypeOfferings" - - "ec2:DescribeAvailabilityZones" - - "ssm:GetParameter" - -fargateProfiles: - - name: karpenter - selectors: - - namespace: karpenter - - name: kube-system - selectors: - - namespace: kube-system diff --git a/pkg/cloudprovider/aws/docs/examples/provisioner.yaml b/pkg/cloudprovider/aws/docs/examples/provisioner.yaml deleted file mode 100644 index 1bddd65ecdca..000000000000 --- a/pkg/cloudprovider/aws/docs/examples/provisioner.yaml +++ /dev/null @@ -1,31 +0,0 @@ -apiVersion: karpenter.sh/v1alpha3 -kind: Provisioner -metadata: - name: default -spec: - # Provisioned nodes connect to this cluster - cluster: - name: "${CLUSTER_NAME}" - caBundle: "${CLUSTER_CA_BUNDLE}" - endpoint: "${CLUSTER_ENDPOINT}" - - # If nil, the feature is disabled, nodes will never expire - ttlSecondsUntilExpired: 2592000 # 30 Days = 60 * 60 * 24 * 30 Seconds; - - # If nil, the feature is disabled, nodes will never scale down due to low utilization - ttlSecondsAfterEmpty: 30 - - # Provisioned nodes will have these taints - taints: - - key: example.com/special-taint - effect: NoSchedule - - # Provisioned nodes will have these labels - labels: - ##### AWS Specific ##### - # Constrain node launch template, default="bottlerocket" - node.k8s.aws/launch-template-id: "bottlerocket-qwertyuiop" - # Constrain node launch template, default="$LATEST" - node.k8s.aws/launch-template-version: "my-special-version" - # Constrain node capacity type, default="on-demand" - node.k8s.aws/capacity-type: "spot" diff --git a/website/content/en/docs/getting-started/_index.md b/website/content/en/docs/getting-started/_index.md index 2ccb3ff089ad..09d8046bba69 100644 --- a/website/content/en/docs/getting-started/_index.md +++ b/website/content/en/docs/getting-started/_index.md @@ -10,11 +10,11 @@ menu: Karpenter automatically provisions new nodes in response to unschedulable pods. Karpenter does this by observing events within the Kubernetes cluster, -and then sending commands to the underlying cloud provider. +and then sending commands to the underlying cloud provider. In this example, the cluster is running on Amazon Web Services (AWS) Elastic Kubernetes Service (EKS). Karpenter is designed to be cloud provider agnostic, -but currently only supports AWS. Contributions are welcomed. +but currently only supports AWS. Contributions are welcomed. This guide should take less than 1 hour to complete, and cost less than $0.25. Follow the clean-up instructions to reduce any charges. @@ -25,7 +25,7 @@ Karpenter is installed in clusters with a helm chart. Karpenter additionally requires IAM Roles for Service Accounts (IRSA). IRSA permits Karpenter (within the cluster) to make privileged requests to AWS (as -the cloud provider). +the cloud provider). ### Required Utilities @@ -36,56 +36,47 @@ Install these tools before proceeding: 3. `eksctl` - [the CLI for AWS EKS](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) Login to the AWS CLI with a user that has sufficient privileges to create a -cluster. +cluster. ### Environment Variables After setting up the tools, set the following environment variables to store -commonly used values. +commonly used values. ```bash export CLUSTER_NAME=$USER-karpenter-demo export AWS_DEFAULT_REGION=us-west-2 AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) -KARPENTER_VERSION=$(curl -fsSL \ - https://api.github.com/repos/awslabs/karpenter/releases/latest \ - | jq -r '.tag_name') ``` ### Create a Cluster -Create a cluster with `eksctl`. The [example configuration](eks-config.yaml) file specifies a basic cluster (name, region), and an IAM role for Karpenter to use. +Create a cluster with `eksctl`. The [example configuration](eks-config.yaml) file specifies a basic cluster (name, region), and an IAM role for Karpenter to use. ```bash -curl -fsSL https://www.karpenter.sh/docs/getting-started/eks-config.yaml \ +curl -fsSL https://karpenter.sh/docs/getting-started/eksctl.yaml \ | envsubst \ | eksctl create cluster -f - ``` -This guide uses a regular (un-managed) node group to host Karpenter. +This guide uses a self-managed node group to host Karpenter. -Karpenter itself can run anywhere, including on self-managed node groups, [managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html), or [AWS Fargate](https://aws.amazon.com/fargate/). +Karpenter itself can run anywhere, including on [self-managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/worker.html), [managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html), or [AWS Fargate](https://aws.amazon.com/fargate/). -Karpenter will provision new traditional instances on EC2. +Karpenter will provision EC2 instances in your account. -Additionally, the configuration file sets up an [OIDC -provider](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens), -necessary for IRSA (see below). Kubernetes supports OIDC as a standardized way -of communicating with identity providers. +Additionally, the configuration file sets up [IAM Roles for Service Accounts](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html) (IRSA), which grants Karpenter permissions like launching instances. -### Setup Authentication from Kubernetes to AWS (IRSA) +### Setup an IAM InstanceProfile for your Nodes -IAM Roles for Service Accounts (IRSA) maps Kubernetes resources to roles -(permission sets) on AWS. +Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking. Karpenter discovers the InstanceProfile using the name `KarpenterNodeRole-${ClusterName}`. -First, define a role using the template below. It provides full access to EC2, -and limited access to other services such as EKS and Elastic Container Registry -(ECR). +First, create the IAM resources using AWS CloudFormation. ```bash # Creates IAM resources used by Karpenter TEMPOUT=$(mktemp) -curl -fsSL https://www.karpenter.sh/docs/getting-started/karpenter.cloudformation.yaml > $TEMPOUT \ +curl -fsSL https://karpenter.sh/docs/getting-started/cloudformation.yaml > $TEMPOUT \ && aws cloudformation deploy \ --stack-name Karpenter-${CLUSTER_NAME} \ --template-file ${TEMPOUT} \ @@ -93,7 +84,7 @@ curl -fsSL https://www.karpenter.sh/docs/getting-started/karpenter.cloudformatio --parameter-overrides ClusterName=${CLUSTER_NAME} ``` -Second, create the mapping between Kubernetes resources and the new IAM role. +Second, grant access to instances using the profile to connect to the cluster. ```bash # Add the Karpenter node role to your aws-auth configmap, allowing nodes with this role to connect to the cluster. @@ -105,11 +96,11 @@ eksctl create iamidentitymapping \ --group system:nodes ``` -Now, Karpenter can send requests for new EC2 instances to AWS and those instances can connect to your cluster. +Now, Karpenter can launch new EC2 instances and those instances can connect to your cluster. ### Install Karpenter Helm Chart -Use helm to deploy Karpenter to the cluster. +Use helm to deploy Karpenter to the cluster. We created a Kubernetes service account when we created the cluster using eksctl. Thus, we don't need the helm chart to do that. @@ -124,22 +115,19 @@ helm upgrade --install karpenter karpenter/karpenter \ ### Provisioner A single Karpenter provisioner is capable of handling many different pod -shapes. In other words, Karpenter eliminates the need to manage many different -node groups. Karpenter makes scheduling and provisioning decisions based on pod -attributes such as labels and affinity. +shapes. Karpenter makes scheduling and provisioning decisions based on pod +attributes such as labels and affinity. In other words, Karpenter eliminates +the need to manage many different node groups. -Create a simple default provisioner using the command below. This provisioner -provides instances with the default certificate bundle, and the control plane -endpoint url. +Create a default provisioner using the command below. This provisioner +configures instances to connect to your cluster's endpoint and discovers +resources like subnets and security groups using the cluster's name. -Importantly, the `ttlSecondsAfterEmpty` value configures Karpenter to -deprovision empty nodes. This behavior can be disabled by leaving the value -undefined. - -Review the [provsioner CRD](/docs/provisioner-crd) for more information. For example, -`ttlSecondsUntilExpired` configures Karpenter to deprovision -nodes when a maximum age is reached. +The `ttlSecondsAfterEmpty` value configures Karpenter to terminate empty nodes. +This behavior can be disabled by leaving the value undefined. +Review the [provsioner CRD](/docs/provisioner-crd) for more information. For example, +`ttlSecondsUntilExpired` configures Karpenter to terminate nodes when a maximum age is reached. ```bash cat <