launchtemplate docs

website/content/en/docs/cloud-providers/AWS/_index.md

@@ -0,0 +1,5 @@
+---
+title: "AWS"
+linkTitle: "AWS"
+weight: 70
+---

website/content/en/docs/cloud-providers/AWS.md → website/content/en/docs/cloud-providers/AWS/aws-spec-fields.md

@@ -1,13 +1,11 @@
 ---
-title: "Amazon Web Services (AWS)"
-linkTitle: "AWS"
+title: "Specifying Values to Control AWS Provisioning"
+linkTitle: "Spec Fields"
 weight: 10
 ---
 
-## Control Provisioning with Labels
-
 The [Provisioner CRD]({{< ref "provisioner-crd.md" >}}) supports defining
-node properties like instance type and zone.For certain well-known labels (documented below), Karpenter will provision
+node properties like instance type and zone. For certain well-known labels (documented below), Karpenter will provision
 nodes accordingly. For example, in response to a label of
 `topology.kubernetes.io/zone=us-east-1c`, Karpenter will provision nodes in
 that availability zone.

website/content/en/docs/cloud-providers/AWS/launch-templates.md

@@ -0,0 +1,233 @@
+---
+title: "Launch Templates and Custom Images"
+linkTitle: "Launch Templates"
+weight: 80
+---
+
+By default, Karpenter generates launch templates that use [EKS Optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template. 
+
+Note: By customizing the image, you are taking responsibility for maintaining the image, including security updates. In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS. 
+
+
+## Introduction
+
+Karpenter follows existing AWS patterns for customizing the base image of
+instances. More specifically, Karpenter uses [EC2 launch templates](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html). Launch
+templates may specify many values. The pivotal value is the base image (AMI).
+Launch templates further specify many different parameters related to networking, authorization, instance type, and more. 
+
+Launch Templates and AMIs are unique to AWS regions, similar to EKS clusters. IAM resources are global. 
+
+**Karpenter only implementes a subset of launch template fields, and some fields should not be set.**  
+
+This guide describes requirements for using launch templates with Karpenter, and later an example procedure.
+
+## Launch Template Requirements
+
+The Launch Template resource includes a large number of fields. AWS accepts launch templates with any subset of these fields defined.
+
+Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile. 
+
+Finally, **the majority of Launch Template fields should not be set** (or will have no effect), such as network interfaces and instance type. 
+
+## Important Fields
+
+When creating a custom launch template, the AMI and User Data are the defining characteristics. Instance Profile (IAM Role) and Security Group (firewall rules) are also important for Karpenter. 
+
+### AMI
+
+AMI (Amazon Machine Image), is the base image/VM for a launch template.
+
+[Review the instructions for importing a VM to AWS.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html) Note the AMI id generated by this process, such as, 
+`ami-074cce78125f09d61`.
+
+### User Data - Autoconfigure
+
+Importantly, the AMI must support automatically connecting to a cluster based
+on "user data", or a base64 encoded string passed to the instance at startup.
+The syntax and purpose of the user data varies between images. The Karpenter
+default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands). 
+
+[AWS calls data passed to an instance at launch time "user
+data".](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts)
+
+In the default configuration, Karpenter uses an EKS optimized version of AL2 and passes the hostname of the Kubernetes API server, and a certificate. The EKS Optimized AMI includes a `bootstrap.sh` script which connects the instance to the cluster, based on the passed data. 
+
+Alternatively, you may reference AWS's [`bootstrap.sh`
+file](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh)
+when building a custom base image. 
+
+```
+#!/bin/bash
+/etc/eks/bootstrap.sh <my-cluster-name> \
+--kubelet-extra-args <'--max-pods=40'> \
+--b64-cluster-ca <certificateAuthority> \
+--apiserver-endpoint <endpoint> \
+--dns-cluster-ip <serviceIpv4Cidr> \
+--use-max-pods false
+```
+
+Note, you must populate this command with live values. Karpenter will
+not change the user data in the launch template. 
+
+Encode using yaml function `!Base64` yaml function or `cat userdata.sh | base64 > userdata-encoded.txt` shell command. 
+
+**Bootstrap Script Parameters**
+
+The sample bootstrap script requires information to join the cluster.
+
+These values may be found using:
+```
+aws eks describe-cluster --name MyKarpenterCluster 
+```
+
+**Kubelet Arguments**
+
+Specifying max-pods can break Karpenter's binpacking logic (it has no way to know what this setting is). If Karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule.
+
+## Situational Fields 
+
+Configure these values in response to a particular use case, such as nodes interacting with another AWS service, or using EBS storage on the node. 
+
+### Instance Profile - IAM
+
+The launch template must include an "instance profile" -- an IAM role. 
+
+The instance profile must include *at least* the permissions of the default Karpenter node instance profile. See the default role, `KarpenterNodeRole`, in the full example below for more information. 
+
+See also, [the managed policy "AmazonEKSWorkerNodePolicy"](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEKSWorkerNodePolicy) which includes permission to describe clusters and subnets.
+
+### Storage
+
+Karpenter expects nothing of node storage. Configure as needed for your base
+image.
+
+### Security Groups - Firewall
+
+The launch template may include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. If none is specified, the default security group of the cluster VPC is used. 
+
+The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet's network access control list (network ACL). 
+
+## Fields with Undefined Behavior
+
+Resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template. 
+
+### Instance Type
+
+The instance type should not be specified in the launch template. Karpenter
+will determine the launch template at run time. 
+
+### Network Interfaces
+
+The [AWS CNI](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) will configure the network interfaces. Do not configure network instances in the launch template.
+
+## Creating the Launch Template
+
+Launch Templates may be created via the web console, the AWS CLI, or
+CloudFormation. 
+
+### CloudFormation
+
+The procedure, in summary, is to:
+1. [Create an AMI as described in the EC2 documentation.](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html)
+2. Write a EC2 Launch Template specification including the AMI.
+3. Push the specification to AWS with CloudFormation.
+4. Update the Provisioner CRD to specify the new Launch Template.
+
+An example yaml cloudformation definition of a launch template for Karpenter is
+provided below. 
+
+CloudFormation yaml is suited for the moderately high configuration density of
+launch templates, and creating the unusual InstanceProfile resource. 
+
+You must manually replace these values in the template:
+- SecurityGroupID
+  - list all security groups with `aws ec2 describe-security-groups`
+- Parameters in UserData
+- AMI
+
+```yaml
+AWSTemplateFormatVersion: '2010-09-09'
+Resources:
+  # create InstanceProfile wrapper on NodeRole
+  KarpenterNodeInstanceProfile:
+    Type: "AWS::IAM::InstanceProfile"
+    Properties:
+      InstanceProfileName: "KarpenterNodeInstanceProfile"
+      Path: "/"
+      Roles:
+        - Ref: "KarpenterNodeRole"
+  # create role with basic permissions for EKS node
+  KarpenterNodeRole:
+    Type: "AWS::IAM::Role"
+    Properties:
+      RoleName: "KarpenterNodeRole"
+      Path: /
+      AssumeRolePolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service:
+                !Sub "ec2.${AWS::URLSuffix}"
+            Action:
+              - "sts:AssumeRole"
+      ManagedPolicyArns:
+        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy"
+        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
+  MyLaunchTemplate:
+    Type: AWS::EC2::LaunchTemplate
+    Properties: 
+      LaunchTemplateData: 
+        IamInstanceProfile:
+          # Get ARN of InstanceProfile defined above
+          Arn: !GetAtt
+            - KarpenterNodeInstanceProfile
+            - Arn
+        ImageId: ami-074cce78125f09d61
+        # UserData is Base64 Encoded
+        UserData: !Base64 > 
+            #!/bin/bash
+            /etc/eks/bootstrap.sh 'MyClusterName' \
+            --kubelet-extra-args '--node-labels=node.k8s.aws/capacity-type=spot' \
+            --b64-cluster-ca 'LS0t....0tCg==' \
+            --apiserver-endpoint 'https://B0385BE29EA792E811CB5866D23C856E.gr7.us-east-2.eks.amazonaws.com' 
+        BlockDeviceMappings: 
+          - Ebs:
+              VolumeSize: 80
+              VolumeType: gp3
+            DeviceName: /dev/xvda
+        # The SecurityGroup must be associated with the cluster VPC
+        SecurityGroupIds:
+          - sg-a69adfdb
+      LaunchTemplateName: KarpenterCustomLaunchTemplate
+```
+
+Create the Launch Template by uploading the CloudFormation yaml file. The
+sample yaml creates an IAM Object (InstanceProfile), so `--capabilities
+CAPABILITY_NAMED_IAM` must be indicated.
+
+```
+aws cloudformation create-stack \
+  --stack-name KarpenterLaunchTemplateStack \
+  --template-body file://$(pwd)/lt-cfn-demo.yaml \
+  --capabilities CAPABILITY_NAMED_IAM
+```
+
+### Define LaunchTemplate for Provisioner
+
+The LaunchTemplate is ready to be used. Specify it by name in the [Provisioner
+CRD](provisioner-crd.md). Karpenter will use this template when creating new instances.
+
+```yaml
+apiVersion: karpenter.sh/v1alpha5
+kind: Provisioner
+spec:
+  provider:
+    launchTemplate: CustomKarpenterLaunchTemplateDemo
+
+```
+
+