-
Notifications
You must be signed in to change notification settings - Fork 983
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Refactor Helm chart #1145
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,16 @@ | ||
| apiVersion: v2 | ||
| appVersion: "0.6.1" | ||
| name: karpenter | ||
| description: A Helm chart for https://github.com/aws/karpenter/. | ||
| description: A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes. | ||
| type: application | ||
| version: "0.6.1" | ||
| version: 0.6.1 | ||
| appVersion: 0.6.1 | ||
| keywords: | ||
| - cluster | ||
| - node | ||
| - scheduler | ||
| - autoscaling | ||
| - lifecycle | ||
| home: https://karpenter.sh/ | ||
| icon: https://repository-images.githubusercontent.com/278480393/dab059c8-caa1-4b55-aaa7-3d30e47a5616 | ||
| sources: | ||
| - https://github.com/aws/karpenter/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,56 +1,77 @@ | ||
| # karpenter | ||
|
|
||
| A Helm chart for https://github.com/aws/karpenter/. | ||
| A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes. | ||
|
|
||
|    | ||
|
|
||
| ## Documentation | ||
|
|
||
| For full Karpenter documentation please checkout [https://karpenter.sh](https://karpenter.sh/v0.6.1/). | ||
|
|
||
| ## Installing the Chart | ||
|
|
||
| To install the chart with the release name `karpenter`: | ||
| Before the chart can be installed the repo needs to be added to Helm, run the following commands to add the repo. | ||
|
|
||
| ```console | ||
| $ helm repo add karpenter https://charts.karpenter.sh | ||
| $ helm repo update | ||
| $ helm upgrade --install karpenter karpenter/karpenter --namespace karpenter \ | ||
| --create-namespace --set serviceAccount.create=false --version 0.6.1 \ | ||
| --set controller.clusterName=${CLUSTER_NAME} \ | ||
| --set controller.clusterEndpoint=$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output json) \ | ||
| --wait # for the defaulting webhook to install before creating a Provisioner | ||
| ```bash | ||
| helm repo add karpenter https://charts.karpenter.sh/ | ||
| helm repo update | ||
| ``` | ||
|
|
||
| You can follow the detailed installation instruction [here](https://karpenter.sh/docs/getting-started/#install). | ||
| You can follow the detailed installation instruction in the [documentation](https://karpenter.sh/v0.6.1/getting-started/#install) which covers the Karpenter prerequisites and installation options. The outcome of these instructions should result in something like the following command. | ||
|
|
||
| ```bash | ||
| helm upgrade --install --namespace karpenter --create-namespace \ | ||
| karpenter karpenter/karpenter \ | ||
| --version 0.6.1 \ | ||
| --set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN} | ||
| --set clusterName=${CLUSTER_NAME} \ | ||
| --set clusterEndpoint=${CLUSTER_ENDPOINT} \ | ||
| --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \ | ||
| --wait # for the defaulting webhook to install before creating a Provisioner | ||
| ``` | ||
|
|
||
| ## Values | ||
|
|
||
| | Key | Type | Default | Description | | ||
| |-----|------|---------|-------------| | ||
| | additionalLabels | object | `{}` | Additional labels to add into metadata | | ||
| | additionalAnnotations | object | `{}` | Additional annotations to add into metadata. | | ||
| | additionalLabels | object | `{}` | Additional labels to add into metadata. | | ||
| | affinity | object | `{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"key":"karpenter.sh/provisioner-name","operator":"DoesNotExist"}]}}}` | Affinity rules for scheduling the pod. | | ||
| | aws.defaultInstanceProfile | string | `""` | The default instance profile to use when launching nodes on AWS | | ||
| | controller.affinity | object | `{}` | Affinity rules for scheduling | | ||
| | controller.clusterEndpoint | string | `""` | Cluster endpoint | | ||
| | controller.clusterName | string | `""` | Cluster name | | ||
| | controller.env | list | `[]` | Additional environment variables to run with | | ||
| | controller.image | string | `"public.ecr.aws/karpenter/controller:v0.6.1@sha256:5a0bd78e2f7ada324677e2eb82e53b648593e9de1acf0a8fc84138a1a6be753c"` | Image to use for the Karpenter controller | | ||
| | controller.nodeSelector | object | `{}` | Node selectors to schedule to nodes with labels. | | ||
| | controller.replicas | int | `1` | | | ||
| | controller.resources.limits.cpu | int | `1` | | | ||
| | controller.resources.limits.memory | string | `"1Gi"` | | | ||
| | controller.resources.requests.cpu | int | `1` | | | ||
| | controller.resources.requests.memory | string | `"1Gi"` | | | ||
| | controller.tolerations | list | `[]` | Tolerations to schedule to nodes with taints. | | ||
| | serviceAccount.annotations | object | `{}` | Annotations to add to the service account (like the ARN of the IRSA role) | | ||
| | serviceAccount.create | bool | `true` | Create a service account for the application controller | | ||
| | serviceAccount.name | string | `"karpenter"` | Service account name | | ||
| | webhook.affinity | object | `{}` | Affinity rules for scheduling | | ||
| | webhook.env | list | `[]` | List of environment items to add to the webhook | | ||
| | webhook.hostNetwork | bool | `false` | Set to true if using custom CNI on EKS | | ||
| | webhook.image | string | `"public.ecr.aws/karpenter/webhook:v0.6.1@sha256:7d75747caeb1ca63da1d68925b961c7a61f40faa76aa678320b2d3e090d1713f"` | Image to use for the webhook | | ||
| | webhook.nodeSelector | object | `{}` | Node selectors to schedule to nodes with labels. | | ||
| | webhook.port | int | `8443` | | | ||
| | webhook.replicas | int | `1` | | | ||
| | webhook.resources.limits.cpu | string | `"100m"` | | | ||
| | webhook.resources.limits.memory | string | `"50Mi"` | | | ||
| | webhook.resources.requests.cpu | string | `"100m"` | | | ||
| | webhook.resources.requests.memory | string | `"50Mi"` | | | ||
| | webhook.tolerations | list | `[]` | Tolerations to schedule to nodes with taints. | | ||
| | clusterEndpoint | string | `""` | Cluster endpoint. | | ||
| | clusterName | string | `""` | Cluster name. | | ||
| | controller.env | list | `[]` | Additional environment variables for the controller pod. | | ||
| | controller.image | string | `"public.ecr.aws/karpenter/controller:v0.6.1@sha256:5a0bd78e2f7ada324677e2eb82e53b648593e9de1acf0a8fc84138a1a6be753c"` | Controller image. | | ||
| | controller.logLevel | string | `""` | Controller log level, defaults to the global log level | | ||
| | controller.resources | object | `{"limits":{"cpu":1,"memory":"1Gi"},"requests":{"cpu":1,"memory":"1Gi"}}` | Resources for the controller pod. | | ||
| | controller.securityContext | object | `{}` | SecurityContext for the controller container. | | ||
| | fullnameOverride | string | `""` | Overrides the chart's computed fullname. | | ||
| | hostNetwork | bool | `false` | Bind the pod to the host network. This is required when using a custom CNI. | | ||
| | imagePullPolicy | string | `"IfNotPresent"` | Image pull policy for Docker images. | | ||
| | imagePullSecrets | list | `[]` | Image pull secrets for Docker images. | | ||
| | logLevel | string | `"info"` | Global log level | | ||
| | nameOverride | string | `""` | Overrides the chart's name. | | ||
| | nodeSelector | object | `{"kubernetes.io/os":"linux"}` | Node selectors to schedule the pod to nodes with labels. | | ||
| | podAnnotations | object | `{}` | Additional annotations for the pod. | | ||
| | podLabels | object | `{}` | Additional labels for the pod. | | ||
| | podSecurityContext | object | `{"fsGroup":1000}` | SecurityContext for the pod. | | ||
| | priorityClassName | string | `"system-cluster-critical"` | PriorityClass name for the pod. | | ||
| | replicas | int | `1` | Number of replicas. | | ||
| | serviceAccount.annotations | object | `{}` | Additional annotations for the ServiceAccount. | | ||
| | serviceAccount.create | bool | `true` | Specifies if a ServiceAccount should be created. | | ||
| | serviceAccount.name | string | `""` | The name of the ServiceAccount to use. If not set and create is true, a name is generated using the fullname template. | | ||
| | serviceMonitor.additionalLabels | object | `{}` | Additional labels for the ServiceMonitor. | | ||
| | serviceMonitor.enabled | bool | `false` | Specifies whether a ServiceMonitor should be created. | | ||
| | serviceMonitor.endpointConfig | object | `{}` | Endpoint configuration for the ServiceMonitor. | | ||
| | strategy | object | `{"type":"Recreate"}` | Strategy for updating the pod. | | ||
| | terminationGracePeriodSeconds | string | `nil` | Override the default termination grace period for the pod. | | ||
| | tolerations | list | `[]` | Tolerations to allow the pod to be scheduled to nodes with taints. | | ||
| | webhook.env | list | `[]` | Additional environment variables for the webhook pod. | | ||
| | webhook.image | string | `"public.ecr.aws/karpenter/webhook:v0.6.1@sha256:7d75747caeb1ca63da1d68925b961c7a61f40faa76aa678320b2d3e090d1713f"` | Webhook image. | | ||
| | webhook.logLevel | string | `""` | Webhook log level, defaults to the global log level | | ||
| | webhook.port | int | `8443` | The container port to use for the webhook. | | ||
| | webhook.resources | object | `{"limits":{"cpu":"100m","memory":"50Mi"},"requests":{"cpu":"100m","memory":"50Mi"}}` | Resources for the webhook pod. | | ||
| | webhook.securityContext | object | `{}` | SecurityContext for the webhook container. | | ||
|
|
||
| ---------------------------------------------- | ||
| Autogenerated from chart metadata using [helm-docs v1.7.0](https://github.com/norwoodj/helm-docs/releases/v1.7.0) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: {{ include "karpenter.fullname" . }} | ||
| labels: | ||
| {{- include "karpenter.labels" . | nindent 4 }} | ||
| {{- with .Values.additionalAnnotations }} | ||
| annotations: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| rules: | ||
| - apiGroups: ["karpenter.sh"] | ||
| resources: ["provisioners"] | ||
| verbs: ["get", "list", "watch"] | ||
| - apiGroups: ["karpenter.sh"] | ||
| resources: ["provisioners/status"] | ||
| verbs: ["create", "delete", "patch", "get", "list", "watch"] | ||
| - apiGroups: [""] | ||
| resources: ["persistentvolumes", "persistentvolumeclaims"] | ||
| verbs: ["get", "list", "watch", "update"] | ||
| - apiGroups: ["storage.k8s.io"] | ||
| resources: ["storageclasses"] | ||
| verbs: ["get", "list", "watch"] | ||
| - apiGroups: [""] | ||
| resources: ["nodes", "pods"] | ||
| verbs: ["get", "list", "watch", "patch", "delete"] | ||
| - apiGroups: [""] | ||
| resources: ["configmaps"] | ||
| verbs: ["get", "list", "watch"] | ||
| - apiGroups: [""] | ||
| resources: ["nodes"] | ||
| verbs: ["create"] | ||
| - apiGroups: [""] | ||
| resources: ["pods/binding", "pods/eviction"] | ||
| verbs: ["create"] | ||
| - apiGroups: ["apps"] | ||
| resources: ["daemonsets"] | ||
| verbs: ["list", "watch"] | ||
| - apiGroups: ["admissionregistration.k8s.io"] | ||
| resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"] | ||
| verbs: ["get", "watch", "list", "update"] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this was set to prevent the service account annotated for IRSA to be deleted in the dev workflow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't the annotation be passed in with
--setflag? Otherwise you're using the namespace default Service Account and giving it all the access. I guess for development it doesn't really matter, but this pattern was also copied everywhere as is often the case.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We thought users would be using the getting started guide eksctl command to create the IAM Role for Service Accounts, which creates the service account and annotates with the role arn (https://karpenter.sh/v0.5.5/getting-started/#create-the-karpentercontroller-iam-role). That might be a bad assumption for production installs, I'm not sure. But for dev installs, it's probably accurate and so leaving the
--set serviceAccount.controller.create=falseseems like the best dev workflow experience to me. Otherwise, we'd need some way of persisting the role arn in an env var or something and then using a set to annotate every time wemake apply.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That guide really needs re-writing to promote better practice, although from memory one of the other tabs has a better way of creating the roles and getting the ARN back? It's idiomatic for a Helm chart to create it's own service account(s), so I'd suggest documenting the externally created one, but with a name specified so it's not the default namespace one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the dev workflow, once the role has been created it persists outside of the K8s resources. It'd be simplest to set it in the env.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The docs that need updating are here (I think that's all of the places):
--role-onlyto the eksctl cmd to prevent creating the service account too)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bwagner5 shouldn't we be waiting until there is a release to make the changes to the docs for that release? The
v0.6.0chart has already been released so I don't think the docs shouldn't change for that version? This is one of the problems of directly linking your chart releases to your package releases.While I wait for a response I'll update the non-docs references and undo the change to docs I accidentally made previously.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bwagner5 @ellistarn how do the docs get updated for a release?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All docs before a release go in the preview directory https://github.com/aws/karpenter/tree/main/website/content/en/preview
When we cut a release, the preview directory is copied to the version we are releasing, so it's fine to make the doc updates to preview in this PR. We shouldn't ever modify a specific version docs outside of a release or a bug fix in the docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These should be updated now.