Need clear installation documentation

[dbbc96]

Description

So i looking at your documentation on installation (specifically on EKS in AWS), i see some gaps in what i feel need to addressed. Some of the links pointing to AWS Blueprints are either broken or provide little help. What i would like to see is clearly the steps on installing Karpenter on an existing cluster. For example step 1) tagging the resources (and go into that detail) 2) Setup IAM roles and policies (go into this as there is no documentation that i saw that talks about needing SQS. however it looks to be needed or you will get errors from the logs) 3) install CRDs and helm stuff along with authmap stuff.

Most of the steps i have seen are for users when migrating from Cluster AutoScaler but if someone isn't doing that then there no reason for a user to click on that. The AWS Blueprints do have a lot of the terraform however jumping between the "addon" and "addons" is a bit much and noisy and has other stuff that isn't needed to just get Karpenter installed.
The cloudformation mentioned in your docs also appears to setup more than needed if the infrastructure is setup and just needs Karpenter.

[njtran]

#4394 should make this better.

[dbbc96]

@njtran i see this is talking about CloudFormation. Not sure if this will only focus on CF setup. If this is the case, its not exactly what i was looking for. Please let me know know. All that i was inquiring about is just the main setup required for Karpenter with the assumption that someone already has a cluster setup but without anything else. So just looking for the steps for each component that Karpenter needs to work correctly. We use Terraform however not expecting the docs to list terraform. It should be the requirements without any assumptions on the method that someone wishes to implement.

[njtran]

From what I understand, the CFN PR will make it easier to understand all the resources you need to create for the Karpenter install, which most users will require. The other bit about tagging should be covered with https://karpenter.sh/docs/getting-started/migrating-from-cas/. @chrisnegus can chime in here, but I think there's an overarching effort to make this better. Which part of the migrating-from-cas guide is not sufficient for you? Can you share which lines/sections in particular?

If you're willing to help contribute to the docs I can work with you on this.

[dbbc96]

my suggestion would be a central place for all the "pre-reqs". the migrating from cas section, is fine for someone who would be wanting to migrate. For someone who never installed cas, a regular user would never go there. What you could do instead is have a page for all the pre-reqs like iam, tagging, sqs, authmap, etc. and then in the cas guide, they reference this new page. Cause the pre-reqs would be required for any user who is new or migrating from cas

[njtran]

Got it. @chrisnegus should be working on this. If not the PR that I linked, he should be able to eventually get to it. @chrisnegus is this the best place to track this effort?

[chrisnegus]

@njtran Yes, let's track this effort here. I think much of what is described here is covered in the new CloudFormation page. But I'll go through and see if any of the other issues brought up still need to be added to the docs.

[dbbc96]

i will wait till you all have finished but one other comment is tool make documentation tool agnostic as not everyone uses cloudformation.

[chrisnegus]

The CloudFormation page doesn't cover everything that @dbbc96 is looking for. I had started work on such a guide (essentially, migrating to Karpenter from any existing cluster). I'll try to cover these requests when I pick up work on that guide again. For now, I added a link from the getting started guide to the CloudFormation page in PR #4997.

[e-koma]

I agree with @dbbc96

the migration guide, Migrating from Cluster Autoscaler, includes instructions on how to install CRDs, but these instructions are absent from the installation guide. This means that people who only look at the Getting Started with Karpenter section will not have enough information.

The migration guide details the CRD installation commands as follows:

kubectl create -f \
    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml"
kubectl create -f \
    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml"
kubectl create -f \
    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml"

However, these are not mentioned in the installation guide.
Correcting the CloudFormation is not sufficient. the CRDs remain uninstalled.

Getting Started with Karpenter should include missing information, such as applying CRDs.

[SebastianScherer88]

@dbbc96 i have tweaked the official terraform karpenter example (removed the fargate hosting of the controller and addressed the service linked role issue) in this directory of my personal repo.

if you want to really drill down into the resource level dependencies (as you thought, the SQS is a requirement/dependency), you can dig into the official terraform karpenter module that's being used in the example.

unfortunately these really useful resources havent made their way into the karpenter docs yet, it seems.

hope that helps!

[billnbell3]

For AWS Fargate it would be good to run karpenter namespace on Fargate nodes for higher reliability.

I tried doing this and it was giving me multiple errors in split configuration. Karpenter pods on Fargate node and my workload on Karpenter nodes. I keep getting API errors and DNS issues.

Maybe good documentation can help ?