From 3cb492341f8043bece07a73c880da22ebd6d28ee Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Wed, 20 Oct 2021 13:29:56 -0500 Subject: [PATCH 01/14] draft launchtemplate docs revise draft on lts revise lt docs wrap lines --- website/content/en/docs/launch-templates.md | 177 ++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 website/content/en/docs/launch-templates.md diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md new file mode 100644 index 000000000000..0079b33813c7 --- /dev/null +++ b/website/content/en/docs/launch-templates.md @@ -0,0 +1,177 @@ +# Custom Images + +## Introduction + +Karpenter follows existing AWS patterns for customizing the base image of +instances. More specifically, Karpenter uses EC2 launch templates. Launch +templates may specify many values. The pivotal value is the base image (AMI). +Launch templates further specify many parameters related to networking, +authorization, instance type, and more. + +## Launch Template Configuration + +### AMI + +Use the AWS CLI to import virtual machine images as AMIs. + +### User Data - Autoconfigure + +Importantly, the AMI must support automatically connecting to a cluster based +on "user data", or a base64 encoded string passed to the instance at startup. +The syntax and purpose of the user data varies between images. The Karpenter +default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). + +[AWS calls data passed to an instance at launch time "user +data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) + +In the default configuration, Karpenter uses AL2 and passes the hostname of the +Kubernetes API server, and a certificate. The instance subsequently uses this +information to securely join the cluster. + +When building a custom image, you may reference AWS's [`bootstrap.sh` +file](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh) +from GitHub, and this associated `user data` startup script. + +``` +#!/bin/bash +/etc/eks/bootstrap.sh \ +--kubelet-extra-args <'--max-pods=40'> \ +--b64-cluster-ca \ +--apiserver-endpoint +--dns-cluster-ip .10 +--use-max-pods false +``` + +Note, you must populate this startup script with live values. Karpenter will +not change the user data in the launch template. + +### Instance Type + +The instance type should not be specified in the launch template. Karpenter +will determine the launch template at run time. + +### Instance Profile - IAM + +The launch template must include an "instance profile" -- a set of IAM roles. + +The instance profile must include all the permissions of the default Karpenter +node instance profile. For example, permission to run containers and manage +networking. See the default role, `KarpenterNodeRole`, in the full example +below for more information. + +### Storage + +Karpenter expects nothing of node storage. Configure as needed for your base +image. + +### Security Groups - Firewall + +EKS configures security groups (i.e., instance firewall rules) automatically. + +However, you may manually specify a security group. The security group must +permit communication with EKS control plane. Outbound access should be +permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and +your subnet's network access control list (network ACL). + +The security group must be associated with the virtual private cloud (VPC) of +the EKS cluster. + +### Network Interfaces + +EKS will configure the network interfaces. Do not configure network instances +in the launch template. + +## Creating the Launch Template + +Launch Templates may be created via the web console, the AWS CLI, or +CloudFormation. + +### CloudFormation + + +An example yaml cloudformation definition of a launch template for Karpenter is +provided below. + +Cloudformation yaml is suited for the moderately high configuration density of +launch templates, and creating the unusual InstanceProfile resource. + +```yaml +AWSTemplateFormatVersion: '2010-09-09' +Resources: + # create InstanceProfile wrapper on NodeRole + KarpenterNodeInstanceProfile: + Type: "AWS::IAM::InstanceProfile" + Properties: + InstanceProfileName: "KarpenterNodeInstanceProfile" + Path: "/" + Roles: + - Ref: "KarpenterNodeRole" + # create role with basic permissions for EKS node + KarpenterNodeRole: + Type: "AWS::IAM::Role" + Properties: + RoleName: "KarpenterNodeRole" + Path: / + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Service: + !Sub "ec2.${AWS::URLSuffix}" + Action: + - "sts:AssumeRole" + ManagedPolicyArns: + - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy" + - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy" + - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" + - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" + MyLaunchTemplate: + Type: AWS::EC2::LaunchTemplate + Properties: + LaunchTemplateData: + IamInstanceProfile: + # Get ARN of InstanceProfile defined above + Arn: !GetAtt + - KarpenterInstanceProfile + - Arn + ImageId: ami-074cce78125f09d61 + # UserData is Base64 Encoded + UserData: "IyEvYmluL2Jhc2gKL2V0Yy9la3MvYm9vdHN0cmFwLnNoIDxteS1jbHVzdGVyLW5hbWU+IFwKLS1rdWJlbGV0LWV4dHJhLWFyZ3MgPCctLW1heC1wb2RzPTQwJz4gXAotLWI2NC1jbHVzdGVyLWNhIDxjZXJ0aWZpY2F0ZUF1dGhvcml0eT4gXAotLWFwaXNlcnZlci1lbmRwb2ludCA8ZW5kcG9pbnQ+IAotLWRucy1jbHVzdGVyLWlwIDxzZXJpdmNlSXB2NENpZHI+LjEwCi0tdXNlLW1heC1wb2RzIGZhbHNl" + BlockDeviceMappings: + - Ebs: + VolumeSize: 80 + VolumeType: gp3 + DeviceName: /dev/xvda + # The SecurityGroup must be associated with the cluster VPC + SecurityGroupIds: + - sg-a69adfdb + LaunchTemplateName: KarpenterCustomLaunchTemplate +``` + +Create the Launch Template by uploading the CloudFormation yaml file. The +sample yaml creates an IAM Object (InstanceProfile), so `--capabilities +CAPABILITY_NAMED_IAM` must be indicated. + +``` +aws cloudformation create-stack \ + --stack-name KarpenterLaunchTemplateStack \ + --template-body file:///Users/gcline/Desktop/lt-cfn-demo.yaml \ + --capabilities CAPABILITY_NAMED_IAM +``` + +### Define LaunchTemplate for Provisioner + +The LaunchTemplate is ready to be used. Specify it by name in the Provisioner +CRD. Karpenter will use this template when creating new instances. + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner +spec: + provider: + launchTemplate: CustomKarpenterLaunchTemplateDemo + +``` + + From f172150f317505639419b1fbd0961f34dc0f676d Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 26 Oct 2021 13:06:46 -0500 Subject: [PATCH 02/14] fix frontmatter --- website/content/en/docs/launch-templates.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md index 0079b33813c7..e9877de64e00 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/launch-templates.md @@ -1,4 +1,10 @@ -# Custom Images +--- +title: "Launch Templates" +linkTitle: "Launch Templates" +weight: 80 +--- + +# Launch Templates and Custom Images ## Introduction From 985c4fcc44becb76831a6ef80866bfd8ed3b86e7 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 26 Oct 2021 13:08:15 -0500 Subject: [PATCH 03/14] refix frontmatter --- website/content/en/docs/launch-templates.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md index e9877de64e00..c1121070f60d 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/launch-templates.md @@ -1,11 +1,9 @@ --- -title: "Launch Templates" +title: "Launch Templates and Custom Images" linkTitle: "Launch Templates" weight: 80 --- -# Launch Templates and Custom Images - ## Introduction Karpenter follows existing AWS patterns for customizing the base image of From de794f244ed79c27a3995ce51d24ccc5bf728c80 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 26 Oct 2021 13:09:47 -0500 Subject: [PATCH 04/14] revise desc of AMIs --- website/content/en/docs/launch-templates.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md index c1121070f60d..9fb12570dcee 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/launch-templates.md @@ -16,7 +16,10 @@ authorization, instance type, and more. ### AMI -Use the AWS CLI to import virtual machine images as AMIs. +AMI (Amazon Machine Image), is the base image/VM for a launch template. + +[Review the instructions for importing a VM to AWS.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html) Note the AMI id generated by this process, such as, +`ami-074cce78125f09d61`. ### User Data - Autoconfigure From dc08abf686af009c162cfadd640e23de866a2f15 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 26 Oct 2021 14:38:26 -0500 Subject: [PATCH 05/14] revise re gh feedback --- website/content/en/docs/launch-templates.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md index 9fb12570dcee..1e23f66ad477 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/launch-templates.md @@ -7,12 +7,14 @@ weight: 80 ## Introduction Karpenter follows existing AWS patterns for customizing the base image of -instances. More specifically, Karpenter uses EC2 launch templates. Launch +instances. More specifically, Karpenter uses [EC2 launch templates](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). Launch templates may specify many values. The pivotal value is the base image (AMI). Launch templates further specify many parameters related to networking, authorization, instance type, and more. -## Launch Template Configuration +This guide describes requirements for using launch templates with Karpenter, and includes an example procedure. + +## Launch Template Requirements ### AMI @@ -31,13 +33,11 @@ default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). [AWS calls data passed to an instance at launch time "user data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) -In the default configuration, Karpenter uses AL2 and passes the hostname of the -Kubernetes API server, and a certificate. The instance subsequently uses this -information to securely join the cluster. +In the default configuration, Karpenter uses an EKS optimized version of AL2 and passes the hostname of the Kubernetes API server, and a certificate. The EKS Optimized AMI includes a `bootstrap.sh` script which connects the instance to the cluster, based on the passed data. -When building a custom image, you may reference AWS's [`bootstrap.sh` +Alternatively, you may reference AWS's [`bootstrap.sh` file](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh) -from GitHub, and this associated `user data` startup script. +when building a custom base image. ``` #!/bin/bash @@ -95,7 +95,6 @@ CloudFormation. ### CloudFormation - An example yaml cloudformation definition of a launch template for Karpenter is provided below. From 86d4fbd2223064cd72a4322611da435723e01bde Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Sun, 31 Oct 2021 22:22:28 -0500 Subject: [PATCH 06/14] revise in progress --- website/content/en/docs/launch-templates.md | 32 ++++++++++++++++++--- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/launch-templates.md index 1e23f66ad477..5f1d7b1afabe 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/launch-templates.md @@ -45,10 +45,22 @@ when building a custom base image. --kubelet-extra-args <'--max-pods=40'> \ --b64-cluster-ca \ --apiserver-endpoint ---dns-cluster-ip .10 +--dns-cluster-ip --use-max-pods false ``` +Encode using yaml function `!Base64` yaml function or `cat hi.yaml | base64 > hi.base` shell command. + +``` +aws eks describe-cluster --name october --region us-west-1 +``` + +use that above command to get these values... +- certificate authority? +- api server? + +- dns cluster ip? default value is ... + Note, you must populate this startup script with live values. Karpenter will not change the user data in the launch template. @@ -101,6 +113,11 @@ provided below. Cloudformation yaml is suited for the moderately high configuration density of launch templates, and creating the unusual InstanceProfile resource. +You must replace: +- SecurityGroupID + - list all security groups with `aws ec2 describe-security-groups` +- Parameters in UserData + ```yaml AWSTemplateFormatVersion: '2010-09-09' Resources: @@ -143,7 +160,14 @@ Resources: - Arn ImageId: ami-074cce78125f09d61 # UserData is Base64 Encoded - UserData: "IyEvYmluL2Jhc2gKL2V0Yy9la3MvYm9vdHN0cmFwLnNoIDxteS1jbHVzdGVyLW5hbWU+IFwKLS1rdWJlbGV0LWV4dHJhLWFyZ3MgPCctLW1heC1wb2RzPTQwJz4gXAotLWI2NC1jbHVzdGVyLWNhIDxjZXJ0aWZpY2F0ZUF1dGhvcml0eT4gXAotLWFwaXNlcnZlci1lbmRwb2ludCA8ZW5kcG9pbnQ+IAotLWRucy1jbHVzdGVyLWlwIDxzZXJpdmNlSXB2NENpZHI+LjEwCi0tdXNlLW1heC1wb2RzIGZhbHNl" + UserData: !Base64 > + #!/bin/bash + /etc/eks/bootstrap.sh \ + --kubelet-extra-args <'--max-pods=40'> \ + --b64-cluster-ca \ + --apiserver-endpoint + --dns-cluster-ip + --use-max-pods false BlockDeviceMappings: - Ebs: VolumeSize: 80 @@ -168,8 +192,8 @@ aws cloudformation create-stack \ ### Define LaunchTemplate for Provisioner -The LaunchTemplate is ready to be used. Specify it by name in the Provisioner -CRD. Karpenter will use this template when creating new instances. +The LaunchTemplate is ready to be used. Specify it by name in the [Provisioner +CRD](provisioner-crd.md). Karpenter will use this template when creating new instances. ```yaml apiVersion: karpenter.sh/v1alpha5 From 4be55f0c5a1af366a96733c68a3f5aa4de762c11 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Thu, 4 Nov 2021 13:22:23 -0500 Subject: [PATCH 07/14] revise lt docs and move under AWS provider --- .../en/docs/cloud-providers/AWS/_index.md | 5 ++ .../{AWS.md => AWS/aws-spec-fields.md} | 8 +-- .../AWS}/launch-templates.md | 57 ++++++++++++++----- 3 files changed, 51 insertions(+), 19 deletions(-) create mode 100644 website/content/en/docs/cloud-providers/AWS/_index.md rename website/content/en/docs/cloud-providers/{AWS.md => AWS/aws-spec-fields.md} (90%) rename website/content/en/docs/{ => cloud-providers/AWS}/launch-templates.md (72%) diff --git a/website/content/en/docs/cloud-providers/AWS/_index.md b/website/content/en/docs/cloud-providers/AWS/_index.md new file mode 100644 index 000000000000..13f1c3f9e722 --- /dev/null +++ b/website/content/en/docs/cloud-providers/AWS/_index.md @@ -0,0 +1,5 @@ +--- +title: "AWS" +linkTitle: "AWS" +weight: 70 +--- \ No newline at end of file diff --git a/website/content/en/docs/cloud-providers/AWS.md b/website/content/en/docs/cloud-providers/AWS/aws-spec-fields.md similarity index 90% rename from website/content/en/docs/cloud-providers/AWS.md rename to website/content/en/docs/cloud-providers/AWS/aws-spec-fields.md index d41a3627b5b8..fe10d4c74283 100644 --- a/website/content/en/docs/cloud-providers/AWS.md +++ b/website/content/en/docs/cloud-providers/AWS/aws-spec-fields.md @@ -1,13 +1,11 @@ --- -title: "Amazon Web Services (AWS)" -linkTitle: "AWS" +title: "Specifying Values to Control AWS Provisioning" +linkTitle: "Spec Fields" weight: 10 --- -## Control Provisioning with Labels - The [Provisioner CRD]({{< ref "provisioner-crd.md" >}}) supports defining -node properties like instance type and zone.For certain well-known labels (documented below), Karpenter will provision +node properties like instance type and zone. For certain well-known labels (documented below), Karpenter will provision nodes accordingly. For example, in response to a label of `topology.kubernetes.io/zone=us-east-1c`, Karpenter will provision nodes in that availability zone. diff --git a/website/content/en/docs/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md similarity index 72% rename from website/content/en/docs/launch-templates.md rename to website/content/en/docs/cloud-providers/AWS/launch-templates.md index 5f1d7b1afabe..49677e392b65 100644 --- a/website/content/en/docs/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -4,6 +4,8 @@ linkTitle: "Launch Templates" weight: 80 --- +By default, Karpenter uses the EKS Optimized version of Amazon Linux 2 (AL2) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. + ## Introduction Karpenter follows existing AWS patterns for customizing the base image of @@ -12,10 +14,20 @@ templates may specify many values. The pivotal value is the base image (AMI). Launch templates further specify many parameters related to networking, authorization, instance type, and more. -This guide describes requirements for using launch templates with Karpenter, and includes an example procedure. +This guide describes requirements for using launch templates with Karpenter, and later an example procedure. ## Launch Template Requirements +The Launch Template resource includes a large number of fields. AWS accepts launch templates with any subset of these fields defined. + +Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile. + +Finally, **the majority of Launch Template fields should not be set** (or will have no effect), such as network interfaces and instance type. + +## Important Fields + +When creating a custom launch template, the AMI and User Data are the defining characteristics. + ### AMI AMI (Amazon Machine Image), is the base image/VM for a launch template. @@ -49,25 +61,27 @@ when building a custom base image. --use-max-pods false ``` -Encode using yaml function `!Base64` yaml function or `cat hi.yaml | base64 > hi.base` shell command. +Note, you must populate this command with live values. Karpenter will +not change the user data in the launch template. + +Encode using yaml function `!Base64` yaml function or `cat userdata.sh | base64 > userdata-encoded.txt` shell command. + +**Bootstrap Script Parameters** +The sample bootstrap script requires information to join the cluster. + +These values may be found using: ``` -aws eks describe-cluster --name october --region us-west-1 +aws eks describe-cluster --name MyKarpenterCluster ``` -use that above command to get these values... -- certificate authority? -- api server? +**Kubelet Arguments** -- dns cluster ip? default value is ... +Specifying max-pods can break Karpenter's binpacking logic (it has no way to know what this setting is). If karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule. -Note, you must populate this startup script with live values. Karpenter will -not change the user data in the launch template. +## Situational Fields -### Instance Type - -The instance type should not be specified in the launch template. Karpenter -will determine the launch template at run time. +Configure these values in response to a particular use case, such as nodes interacting with another AWS service, or using EBS storage on the node. ### Instance Profile - IAM @@ -95,6 +109,15 @@ your subnet's network access control list (network ACL). The security group must be associated with the virtual private cloud (VPC) of the EKS cluster. +## Fields with Undefined Behavior + +These resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template. + +### Instance Type + +The instance type should not be specified in the launch template. Karpenter +will determine the launch template at run time. + ### Network Interfaces EKS will configure the network interfaces. Do not configure network instances @@ -107,13 +130,19 @@ CloudFormation. ### CloudFormation +The procedure, in summary, is to: +1. [Create an AMI as described in the EC2 documentation.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html) +2. Write a EC2 Launch Template specification including the AMI. +3. Push the specification to AWS with CloudFormation. +4. Update the Provisioner CRD to specify the new Launch Template. + An example yaml cloudformation definition of a launch template for Karpenter is provided below. Cloudformation yaml is suited for the moderately high configuration density of launch templates, and creating the unusual InstanceProfile resource. -You must replace: +You must manually replace these values in the template: - SecurityGroupID - list all security groups with `aws ec2 describe-security-groups` - Parameters in UserData From 06273641b588bcae79b5d7f07afad77c4b1b5aae Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Thu, 4 Nov 2021 21:52:45 -0500 Subject: [PATCH 08/14] Update website/content/en/docs/cloud-providers/AWS/launch-templates.md Co-authored-by: Ellis Tarn --- website/content/en/docs/cloud-providers/AWS/launch-templates.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index 49677e392b65..a845a9808bbf 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -4,7 +4,7 @@ linkTitle: "Launch Templates" weight: 80 --- -By default, Karpenter uses the EKS Optimized version of Amazon Linux 2 (AL2) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. +By default, Karpenter generates launch templates that use [EKS Optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. ## Introduction From 258aa7a8616ff6968f1c8c8cf15e92e4303deef2 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 9 Nov 2021 09:29:14 -0600 Subject: [PATCH 09/14] feedback from bwagner --- .../cloud-providers/AWS/launch-templates.md | 32 ++++++++----------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index a845a9808bbf..91160e44334a 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -11,8 +11,9 @@ By default, Karpenter generates launch templates that use [EKS Optimized AMI](ht Karpenter follows existing AWS patterns for customizing the base image of instances. More specifically, Karpenter uses [EC2 launch templates](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). Launch templates may specify many values. The pivotal value is the base image (AMI). -Launch templates further specify many parameters related to networking, -authorization, instance type, and more. +Launch templates further specify many different parameters related to networking, authorization, instance type, and more. + +**Karpenter only implementes a subset of launch template fields, and some fields should not be set.** This guide describes requirements for using launch templates with Karpenter, and later an example procedure. @@ -26,7 +27,7 @@ Finally, **the majority of Launch Template fields should not be set** (or will h ## Important Fields -When creating a custom launch template, the AMI and User Data are the defining characteristics. +When creating a custom launch template, the AMI and User Data are the defining characteristics. Instance Profile (IAM Role) and Security Group (firewall rules) are also important for Karpenter. ### AMI @@ -87,10 +88,9 @@ Configure these values in response to a particular use case, such as nodes inter The launch template must include an "instance profile" -- a set of IAM roles. -The instance profile must include all the permissions of the default Karpenter -node instance profile. For example, permission to run containers and manage -networking. See the default role, `KarpenterNodeRole`, in the full example -below for more information. +The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. + +See also, [the managed policy "AmazonEKSWorkerNodePolicy"](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEKSWorkerNodePolicy) which includes permission to describe clusters and subnets. ### Storage @@ -99,19 +99,13 @@ image. ### Security Groups - Firewall -EKS configures security groups (i.e., instance firewall rules) automatically. - -However, you may manually specify a security group. The security group must -permit communication with EKS control plane. Outbound access should be -permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and -your subnet's network access control list (network ACL). +The launch template must include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. -The security group must be associated with the virtual private cloud (VPC) of -the EKS cluster. +The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). ## Fields with Undefined Behavior -These resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template. +Resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template. ### Instance Type @@ -120,8 +114,7 @@ will determine the launch template at run time. ### Network Interfaces -EKS will configure the network interfaces. Do not configure network instances -in the launch template. +The [AWS CNI](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) will configure the network interfaces. Do not configure network instances in the launch template. ## Creating the Launch Template @@ -146,6 +139,7 @@ You must manually replace these values in the template: - SecurityGroupID - list all security groups with `aws ec2 describe-security-groups` - Parameters in UserData +- AMI ```yaml AWSTemplateFormatVersion: '2010-09-09' @@ -215,7 +209,7 @@ CAPABILITY_NAMED_IAM` must be indicated. ``` aws cloudformation create-stack \ --stack-name KarpenterLaunchTemplateStack \ - --template-body file:///Users/gcline/Desktop/lt-cfn-demo.yaml \ + --template-body file://$(pwd)/lt-cfn-demo.yaml \ --capabilities CAPABILITY_NAMED_IAM ``` From 38ddc3a379299955daff4bfbcdedf7431df879c8 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Tue, 9 Nov 2021 10:53:44 -0600 Subject: [PATCH 10/14] add note on image updates --- .../content/en/docs/cloud-providers/AWS/launch-templates.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index 91160e44334a..24a1fe6cda91 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -6,6 +6,9 @@ weight: 80 By default, Karpenter generates launch templates that use [EKS Optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. +Note: By customizing the image, you are taking responsibility for maintianing the image, including security updates. In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS. + + ## Introduction Karpenter follows existing AWS patterns for customizing the base image of From 43f78894eea49b6e12b1d03017cde77fa740f0ef Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Thu, 11 Nov 2021 21:54:00 -0600 Subject: [PATCH 11/14] Apply suggestions from code review Co-authored-by: Elton --- .../cloud-providers/AWS/launch-templates.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index 24a1fe6cda91..0719eeae14d1 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -26,7 +26,7 @@ The Launch Template resource includes a large number of fields. AWS accepts laun Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile. -Finally, **the majority of Launch Template fields should not be set** (or will have no effect), such as network interfaces and instance type. +Finally, **the majority of Launch Template fields should not be set** (or will have no effect). If it's not documented below, you should assume that the field should not be set. ## Important Fields @@ -44,7 +44,7 @@ AMI (Amazon Machine Image), is the base image/VM for a launch template. Importantly, the AMI must support automatically connecting to a cluster based on "user data", or a base64 encoded string passed to the instance at startup. The syntax and purpose of the user data varies between images. The Karpenter -default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). +default OS, Amazon Linux 2 (AL2), accepts shell scripts (e.g. bash commands). [AWS calls data passed to an instance at launch time "user data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) @@ -68,7 +68,7 @@ when building a custom base image. Note, you must populate this command with live values. Karpenter will not change the user data in the launch template. -Encode using yaml function `!Base64` yaml function or `cat userdata.sh | base64 > userdata-encoded.txt` shell command. +You can encode your user data using yaml function `!Base64` yaml function or shell command `cat userdata.sh | base64 > userdata-encoded.txt` . **Bootstrap Script Parameters** @@ -76,7 +76,8 @@ The sample bootstrap script requires information to join the cluster. These values may be found using: ``` -aws eks describe-cluster --name MyKarpenterCluster +aws eks describe-cluster --name + ``` **Kubelet Arguments** @@ -91,7 +92,8 @@ Configure these values in response to a particular use case, such as nodes inter The launch template must include an "instance profile" -- a set of IAM roles. -The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. +The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the [full example below](#cloudformation) for more information. + See also, [the managed policy "AmazonEKSWorkerNodePolicy"](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEKSWorkerNodePolicy) which includes permission to describe clusters and subnets. @@ -104,7 +106,11 @@ image. The launch template must include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. -The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). +The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: +- HTTPS on port 443 +- DNS (UDP and TCP) on port 53 +- Your subnet's network access control list (network ACL). + ## Fields with Undefined Behavior From 580d24b02e6ffcd08b33fd984aa778b7cbf34d01 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Thu, 11 Nov 2021 21:55:54 -0600 Subject: [PATCH 12/14] typos --- .../cloud-providers/AWS/launch-templates.md | 22 ++++++++----------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index 0719eeae14d1..4efeebd65ec5 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -6,7 +6,7 @@ weight: 80 By default, Karpenter generates launch templates that use [EKS Optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. -Note: By customizing the image, you are taking responsibility for maintianing the image, including security updates. In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS. +Note: By customizing the image, you are taking responsibility for maintaining the image, including security updates. In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS. ## Introduction @@ -16,6 +16,8 @@ instances. More specifically, Karpenter uses [EC2 launch templates](https://docs templates may specify many values. The pivotal value is the base image (AMI). Launch templates further specify many different parameters related to networking, authorization, instance type, and more. +Launch Templates and AMIs are unique to AWS regions, similar to EKS clusters. IAM resources are global. + **Karpenter only implementes a subset of launch template fields, and some fields should not be set.** This guide describes requirements for using launch templates with Karpenter, and later an example procedure. @@ -26,7 +28,7 @@ The Launch Template resource includes a large number of fields. AWS accepts laun Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile. -Finally, **the majority of Launch Template fields should not be set** (or will have no effect). If it's not documented below, you should assume that the field should not be set. +Finally, **the majority of Launch Template fields should not be set** (or will have no effect), such as network interfaces and instance type. ## Important Fields @@ -44,7 +46,7 @@ AMI (Amazon Machine Image), is the base image/VM for a launch template. Importantly, the AMI must support automatically connecting to a cluster based on "user data", or a base64 encoded string passed to the instance at startup. The syntax and purpose of the user data varies between images. The Karpenter -default OS, Amazon Linux 2 (AL2), accepts shell scripts (e.g. bash commands). +default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). [AWS calls data passed to an instance at launch time "user data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) @@ -68,7 +70,7 @@ when building a custom base image. Note, you must populate this command with live values. Karpenter will not change the user data in the launch template. -You can encode your user data using yaml function `!Base64` yaml function or shell command `cat userdata.sh | base64 > userdata-encoded.txt` . +Encode using yaml function `!Base64` yaml function or `cat userdata.sh | base64 > userdata-encoded.txt` shell command. **Bootstrap Script Parameters** @@ -76,8 +78,7 @@ The sample bootstrap script requires information to join the cluster. These values may be found using: ``` -aws eks describe-cluster --name - +aws eks describe-cluster --name MyKarpenterCluster ``` **Kubelet Arguments** @@ -92,8 +93,7 @@ Configure these values in response to a particular use case, such as nodes inter The launch template must include an "instance profile" -- a set of IAM roles. -The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the [full example below](#cloudformation) for more information. - +The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. See also, [the managed policy "AmazonEKSWorkerNodePolicy"](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEKSWorkerNodePolicy) which includes permission to describe clusters and subnets. @@ -106,11 +106,7 @@ image. The launch template must include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. -The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: -- HTTPS on port 443 -- DNS (UDP and TCP) on port 53 -- Your subnet's network access control list (network ACL). - +The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). ## Fields with Undefined Behavior From 6333aa8cc94c3ee0f734a55cda885dbd87549177 Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Fri, 12 Nov 2021 12:17:06 -0600 Subject: [PATCH 13/14] revise desc of security groups --- .../docs/cloud-providers/AWS/launch-templates.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index 4efeebd65ec5..cd89bf0cc0ec 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -104,7 +104,7 @@ image. ### Security Groups - Firewall -The launch template must include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. +The launch template may include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. If none is specified, the default security group of the cluster VPC is used. The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). @@ -184,18 +184,16 @@ Resources: IamInstanceProfile: # Get ARN of InstanceProfile defined above Arn: !GetAtt - - KarpenterInstanceProfile + - KarpenterNodeInstanceProfile - Arn ImageId: ami-074cce78125f09d61 # UserData is Base64 Encoded UserData: !Base64 > #!/bin/bash - /etc/eks/bootstrap.sh \ - --kubelet-extra-args <'--max-pods=40'> \ - --b64-cluster-ca \ - --apiserver-endpoint - --dns-cluster-ip - --use-max-pods false + /etc/eks/bootstrap.sh 'MyClusterName' \ + --kubelet-extra-args '--node-labels=node.k8s.aws/capacity-type=spot' \ + --b64-cluster-ca 'LS0t....0tCg==' \ + --apiserver-endpoint 'https://B0385BE29EA792E811CB5866D23C856E.gr7.us-east-2.eks.amazonaws.com' BlockDeviceMappings: - Ebs: VolumeSize: 80 From ff0a33d196bf99d7169bc798dc16b20976c2e19e Mon Sep 17 00:00:00 2001 From: Geoffrey Cline Date: Mon, 15 Nov 2021 11:00:15 -0600 Subject: [PATCH 14/14] fixup --- .../en/docs/cloud-providers/AWS/launch-templates.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/content/en/docs/cloud-providers/AWS/launch-templates.md b/website/content/en/docs/cloud-providers/AWS/launch-templates.md index cd89bf0cc0ec..3492d404a1dc 100644 --- a/website/content/en/docs/cloud-providers/AWS/launch-templates.md +++ b/website/content/en/docs/cloud-providers/AWS/launch-templates.md @@ -62,8 +62,8 @@ when building a custom base image. /etc/eks/bootstrap.sh \ --kubelet-extra-args <'--max-pods=40'> \ --b64-cluster-ca \ ---apiserver-endpoint ---dns-cluster-ip +--apiserver-endpoint \ +--dns-cluster-ip \ --use-max-pods false ``` @@ -83,7 +83,7 @@ aws eks describe-cluster --name MyKarpenterCluster **Kubelet Arguments** -Specifying max-pods can break Karpenter's binpacking logic (it has no way to know what this setting is). If karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule. +Specifying max-pods can break Karpenter's binpacking logic (it has no way to know what this setting is). If Karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule. ## Situational Fields @@ -91,7 +91,7 @@ Configure these values in response to a particular use case, such as nodes inter ### Instance Profile - IAM -The launch template must include an "instance profile" -- a set of IAM roles. +The launch template must include an "instance profile" -- an IAM role. The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. @@ -137,7 +137,7 @@ The procedure, in summary, is to: An example yaml cloudformation definition of a launch template for Karpenter is provided below. -Cloudformation yaml is suited for the moderately high configuration density of +CloudFormation yaml is suited for the moderately high configuration density of launch templates, and creating the unusual InstanceProfile resource. You must manually replace these values in the template: