-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Creating new Karpenter FAQ #1016
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
3560060
Restoring original FAQ to update
chrisnegus 017ac1b
Merge branch 'main' of github.com:chrisnegus/karpenter into restore-f…
chrisnegus 3d96936
Updated FAQ with new info
chrisnegus 29d3081
Add kOps to list of getting started methods
chrisnegus 42383c4
Fixed kOps link
chrisnegus 5298e5b
Responded to review comments
chrisnegus File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
--- | ||
title: "FAQs" | ||
linkTitle: "FAQs" | ||
weight: 90 | ||
--- | ||
## General | ||
|
||
### How does a provisioner decide to manage a particular node? | ||
chrisnegus marked this conversation as resolved.
Show resolved
Hide resolved
|
||
See [Configuring provisioners]({{< ref "/docs/concepts/#configuring-provisioners" >}}) for information on how Karpenter provisions and manages nodes. | ||
|
||
### What cloud providers are supported? | ||
AWS is the first cloud provider supported by Karpenter, although it is designed to be used with other cloud providers as well. | ||
See [[Cloud provider]({{< ref "/docs/concepts/#cloud-provider" >}}) for details. | ||
|
||
### Can I write my own cloud provider for Karpenter? | ||
Yes, but there is no documentation yet for it. | ||
Start with Karpenter's GitHub [cloudprovider](https://github.com/aws/karpenter/tree/main/pkg/cloudprovider) documentation to see how the AWS provider is built, but there are other sections of the code that will require changes too. | ||
|
||
### What operating system nodes does Karpenter deploy? | ||
By default, Karpenter uses Amazon Linux 2 images. | ||
|
||
### Can I provide my own custom operating system images? | ||
Karpenter allows you to create your own AWS AMIs using custom launch templates. | ||
See [Launch Templates and Custom Images]({{< ref "/docs/aws/launch-templates/" >}}) for details. | ||
|
||
### Can Karpenter deal with workloads for mixed architecture cluster (arm vs. amd)? | ||
Yes. Build and prepare custom arm images as described in [Launch Templates and Custom Images]({{< ref "/docs/aws/launch-templates/" >}}). | ||
Specify the desired architecture when you deploy workloads. | ||
|
||
### What RBAC access is required? | ||
All of the required RBAC rules can be found in the helm chart template. | ||
See the [rbac.yaml](https://github.com/aws/karpenter/blob/main/charts/karpenter/templates/controller/rbac.yaml) file for details. | ||
|
||
### Can I run Karpenter outside of a Kubernetes cluster? | ||
Yes, as long as the controller has network and IAM/RBAC access to the Kubernetes API and your provider API. | ||
|
||
## Compatibility | ||
|
||
### Which versions of Kubernetes does Karpenter support? | ||
Karpenter is tested with Kubernetes v1.19 and later. | ||
|
||
### What Kubernetes distributions are supported? | ||
Karpenter documents integration with a fresh install of the latest AWS Elastic Kubernetes Service (EKS). | ||
Existing EKS distributions can be used, but this use case has not yet been documented. | ||
Other Kubernetes distributions (KOPs, etc.) can be used, but setting up cloud provider permissions for those distributions has not been documented. | ||
|
||
### How does Karpenter interact with AWS node group features? | ||
Provisioners are designed to work alongside static capacity management solutions like EKS Managed Node Groups and EC2 Auto Scaling Groups. | ||
You can manage all capacity using provisioners, use a mixed model with dynamic and statically managed capacity, or use a fully static approach. | ||
We expect most users will use a mixed approach in the near term and provisioner-managed in the long term. | ||
|
||
|
||
### How does Karpenter interact with Kubernetes features? | ||
* Kubernetes Cluster Autoscaler: Karpenter can work alongside cluster autoscaler. | ||
See [Kubernetes cluster autoscaler]({{< ref "/docs/concepts/#kubernetes-cluster-autoscaler" >}}) for details. | ||
* Kubernetes Scheduler: Karpenter focuses on scheduling pods that the Kubernetes scheduler has marked as unschedulable. | ||
See [Scheduling]({{< ref "/docs/concepts/#scheduling" >}}) for details on how Karpenter interacts with the Kubernetes scheduler. | ||
|
||
## Provisioning | ||
### What features does the Karpenter provisioner support? | ||
See [Provisioner API]({{< ref "/docs/provisioner" >}}) for provisioner examples and descriptions of features. | ||
|
||
### Can I create multiple (team-based) provisioners on a cluster? | ||
Yes, provisioners can identify multiple teams based on labels. | ||
See [Provisioner API]({{< ref "/docs/provisioner" >}}) for details. | ||
|
||
### If multiple provisioners are defined, which will my pod use? | ||
|
||
By default, pods will use the rules defined by a provisioner named default. | ||
This is analogous to the default scheduler. | ||
To select an alternative provisioner, use the node selector `karpenter.sh/provisioner-name: alternative-provisioner`. | ||
You must either define a default provisioner or explicitly specify `karpenter.sh/provisioner-name node selector`. | ||
|
||
### Can I set total limits of CPU and memory for a provisioner? | ||
Yes, the setting is provider-specific. | ||
See examples in [Accelerators, GPU]({{< ref "/docs/aws/provisioning/#accelerators-gpu" >}}) Karpenter documentation. | ||
|
||
### Can I mix spot and on-demand EC2 run types? | ||
Yes, see [Example Provisioner Resource]({{< ref "/docs/provisioner/#example-provisioner-resource" >}}) for an example. | ||
|
||
### Can I restrict EC2 instance types? | ||
|
||
* Attribute-based requests are currently not possible. | ||
* You can select instances with special hardware, such as gpu. | ||
|
||
## Workloads | ||
|
||
### How can someone deploying pods take advantage of Karpenter? | ||
|
||
See [Application developer]({{< ref "/docs/concepts/#application-developer" >}}) for descriptions of how Karpenter matches nodes with pod requests. | ||
|
||
### How do I use Karpenter with the AWS load balancer controller? | ||
|
||
* Set the [ALB target type]({{< ref "https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/ingress/annotations/#target-type" >}}) to IP mode for the pods. | ||
Use IP targeting if you want the pods to receive equal weight. | ||
Instance balancing could greatly skew the traffic being sent to a node without also managing host spread of the workload. | ||
* Set [readiness gate]({{< ref "https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/deploy/pod_readiness_gate/" >}}) on the namespace. | ||
The default is round robin at the node level. | ||
For Karpenter, not all nodes are equal. | ||
chrisnegus marked this conversation as resolved.
Show resolved
Hide resolved
|
||
For example, each node will have different performance characteristics and a different number of pods running on it. | ||
A `t3.small` with three instances should not receive the same amount of traffic as a `m5.4xlarge` with dozens of pods. | ||
If you don't specify a spread at the workload level, or limit what instances should be picked, you could get the same amount of traffic sent to the `t3` and `m5`. | ||
|
||
### Can I use Karpenter with EBS disks per availability zone? | ||
Not yet. | ||
|
||
### Can I set `--max-pods` on my nodes? | ||
Not yet. | ||
|
||
## Deprovisioning | ||
### How does Karpenter deprovision nodes? | ||
See [Deprovisioning nodes]({{< ref "/docs/tasks/deprov-nodes" >}}) for information on how Karpenter deprovisions nodes. | ||
|
||
## Upgrading | ||
### How do I upgrade Karpenter? | ||
Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. | ||
Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm seeing a lot of AWS Specific FAQs in here. Can we scope this to AWS?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only 6 of the 27 questions explicitly call out AWS. I'm going to go through all the comments to get this doc in a sane state. Then we can talk through how we might change the scope.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the mean time, any time I find an FAQ question that is duplicated in the concepts section, I'll add a link to it from here. That should help reduce the amount of drift between the two docs.