defaulting.webhook.karpenter.sh not being cleaned after the upgrade from v0.27.0 t0 v0.27.1

[nalshamaajc]

https://github.com/aws/karpenter/blob/2692257998f9b8daec819e13e5d2e886e6c3cd03/charts/karpenter/templates/clusterrole-core.yaml#

I don't see defaulting.webhook.karpenter.sh being created anywhere but it still exists in the clusterrole-core.yaml template, is it there for compatibility?

It is not being cleaned after the upgrade and it's causing errors.

2023-03-30T14:48:31.074Z	ERROR	controller	Reconciler error	{"commit": "7131be2-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "Provisioner": {"name":"default"}, "namespace": "", "name": "default", "reconcileID": "0d9fe71a-effe-450d-a3c3-2ce144c39f77", "error": "Internal error occurred: failed calling webhook \"defaulting.webhook.karpenter.sh\": failed to call webhook: the server rejected our request for an unknown reason"}
2023-03-30T14:48:31.140Z	ERROR	controller	Reconciler error	{"commit": "7131be2-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "Provisioner": {"name":"default"}, "namespace": "", "name": "default", "reconcileID": "3bf2940a-ce39-47a1-b864-09fec9e09aaa", "error": "Internal error occurred: failed calling webhook \"defaulting.webhook.karpenter.sh\": failed to call webhook: the server rejected our request for an unknown reason"}

Solution
kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io/defaulting.webhook.karpenter.sh

Redeploy the Karpenter deployment

[bwagner5]

The permission definitely shouldn't still be in the chart. I've created a PR to remove that. I'm not sure why defaulting.webhook.karpenter.sh didn't get removed when you performed the upgrade.

Are you doing a helm upgrade or are you generating the template and applying it another way?

[nalshamaajc]

no I rely on ArgoCD to do so, and I had the same issue on 2 other clusters I was performing the same operation on where I had to do the same workaround

[jonathan-innis]

no I rely on ArgoCD to do so

Does Argo have functionality to apply through helm upgrade or are you just templating the manifests and applying them through GitOps with Argo?

[nalshamaajc]

In my case it's the latter, not quite sure if there is a feature to force argo to use helm upgrade.

[ellistarn]

Reading this thread, seems like we're good to close. Feel free to reopen if I got this wrong.