Clarify v1 migration

[morremeyer]

Description

How can the docs be improved?

The v1 migration documentation explains in great detail how to manually perform all the upgrade steps.

However, we use ArgoCD to deploy karpenter and the karpenter CRDs, and terraform to manage the IAM policies. Therefore and due to the needed repeatability, performing all these steps manually is not an option for us.

I read through the upgrade guide and came to the following conclusion for any setup that is not managed/upgraded manually on a terminal:

• Steps 1 through 6 are just there to ensure karpenter is running in a version that can be upgraded from
• Steps 7 through 10 are the actual upgrade
  • Step 8 updates the IAM role
  • Steps 9 and 10 update karpenter CRDs and karpenter

The needed steps when not doing everything on a terminal would therefore be:

1. Perform needed steps in https://karpenter.sh/docs/upgrading/v1-migration/#changes-required-before-upgrading-to-v100
2. Update the IAM role used by karpenter (step 8 in the upgrade procedure)
3. Merge the PR that contains the version upgrades (this depends on how you update your versions, we use PRs for that)
4. Update the CRDs
5. Update karpenter

Can someone please confirm this or correct me where I'm wrong? Thanks!

Notes

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jetersen]

There are some lessons learned here from us, early adopters. Here be dragons 😓
#6765

[Vinaum8]

@morremeyer I am following the issues regarding the hook to then test karpenter again and I am also checking if I can change the way I install the CRD.

I use ArgoCD and everything went wrong here hahahaha but, my mistake.

[booleanbetrayal]

There is more discussion around this issue @ #6847

TL;DR - We ended up having to re-package the karpenter Chart (minus crds/) and include karpenter-crd chart in our ArgoCD Application. We also hit issues with Validating and Mutating webhooks post-install, presumably because we had to selectively sync the CRDs manually (and ArgoCD won't run hooks on selective syncs) and there was maybe a post-sync hook to clean these up. Regardless, it was a very trying upgrade. Good luck!

[tvandinther]

Why is the procedure not just:

• Update the IAM role
• Deploy version 1.0

At a later time of convenience...

• Migrate v1beta1 CRs to the v1 API

In the future...

• Safely deploy version 1.x which has v1beta1 CRDs removed

I thought this was the whole reason we versioned APIs in the first place.

Has anyone managed to do this upgrade through Argo CD without making custom charts, running conversion webhooks, reinstalling Karpenter or breaking and redeploying their clusters? I'd love to know. Until then, this upgrade is going to the backlog, v0.36 has been working well enough.

[adrianmiron]

@tvandinther Yeah, this looks like a mess for now.

Using ArgoCD, I managed to get 0.37.2 installed after switching from main chart crds ( disable with skip crd helm option ) to using the karpenter crd chart.

But when going from 0.37.2 to 1.0.1 .... it did not work.

• update crds to v1.0.1, with webhooks enabled - worked
• update main chart to v1.0.1 , did not work, the mutating webhooks are not present in the chart, so the i get errors on that from the crds
• disable the webhooks on the v1.0.1 crd chart and THEN update to v1.0.1 app chart...stil does not work, the crds keep calling th webhooks...somehow

I will be back in a few months when this is all sorted

[Vinaum8]

clearly a problem, even using 2 helm charts.
the helm chart "karpenter": it is not possible to change the service name.
the helm chart "karpenter-crd": it has the correct name, but has a conflict with helm chart karpenter.

The configurations are in conflict.