MutatingWebhookConfiguration Failing on Not Found Error

[seanocca]

Version

Karpenter Version: v0.27.2

Kubernetes Version: v1.24.13-eks-0a21954

Expected Behavior

Not receiving the "not found" error in the logs for a mutatingwebhookconfiguration when it is ending with .aws instead of .io.
Actual webhook name defaulting.webhook.karpenter.k8s.aws
Searching for defaulting.webhook.karpenter.sh

Actual Behavior

Receiving Error in logs
"error retrieving webhook: mutatingwebhookconfiguration.admissionregistration.k8s.io \"defaulting.webhook.karpenter.sh\" not found"

Steps to Reproduce the Problem

Deploy above version to AWS EKS using ArgoCD and let it populate all data

Resource Specs and Logs

karpenter:
  nameOverride: ""
  fullnameOverride: ""
  additionalLabels: {}

  additionalAnnotations: {}
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  serviceAccount:
    create: true
    name: ""
    annotations: {}
  additionalClusterRoleRules: []
  serviceMonitor:
    enabled: false
    additionalLabels: {}
    endpointConfig: {}
  replicas: 1
  revisionHistoryLimit: 10
  strategy:
    rollingUpdate:
      maxUnavailable: 1
  podLabels: {}
  podAnnotations:
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
    karpenter.sh/do-not-evict: "true"
  podDisruptionBudget:
    name: karpenter
    maxUnavailable: 1
  podSecurityContext:
    fsGroup: 1000
  priorityClassName: system-node-critical
  terminationGracePeriodSeconds:
  hostNetwork: false
  dnsPolicy: Default
  dnsConfig: {}
  nodeSelector:
    kubernetes.io/os: linux
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: karpenter.sh/provisioner-name
                operator: DoesNotExist
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: "kubernetes.io/hostname"
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
  tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
  extraVolumes: []
  extraObjects: []

  controller:
    image:
      repository: public.ecr.aws/karpenter/controller
      tag: v0.27.2
    securityContext: {}
    env:
      - name: ENABLE_PROFILING
        value: "true"
    envFrom: []
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 1
        memory: 1Gi
    outputPaths:
      - stdout
    errorOutputPaths:
      - stderr
    logLevel: ""
    logEncoding: ""
    extraVolumeMounts: []
    sidecarContainer: []
    sidecarVolumeMounts: []
    metrics:
      port: 8080
    healthProbe:
      port: 8081
  webhook:
    logLevel: error
    port: 8443
  logLevel: debug
  logEncoding: console
  settings:
    batchMaxDuration: 10s
    batchIdleDuration: 1s
    aws:
      clusterName: "XXXXXXX"
      clusterEndpoint: ""
      defaultInstanceProfile: "XXXXXXXX"
      enablePodENI: false
      enableENILimitedPodDensity: true
      isolatedVPC: false
      nodeNameConvention: "ip-name"
      vmMemoryOverheadPercent: 0.075
      interruptionQueueName: ""
templates
      tags:
    featureGates:
      driftEnabled: false

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bwagner5]

There are actually two webhooks:

• defaulting.webhook.karpenter.k8s.aws (for the AWSNodeTemplate)
• defaulting.webhook.karpenter.sh (for the Provisioner) <- this is actually removed in v0.27.3 (https://karpenter.sh/v0.27/troubleshooting/#failed-calling-webhook-defaultingwebhookkarpentersh)

Are you using the v0.27.2 version of the Karpenter helm chart?

There is a common problem when using argoCD to deploy Karpenter. There seems to be a bug in argo where old webhooks are not cleaned up. These issues are related: #3673 #1971

[seanocca]

There are actually two webhooks:

* defaulting.webhook.karpenter.k8s.aws (for the AWSNodeTemplate)

* defaulting.webhook.karpenter.sh (for the Provisioner) <- this is actually removed in v0.27.3 (https://karpenter.sh/v0.27/troubleshooting/#failed-calling-webhook-defaultingwebhookkarpentersh)

Are you using the v0.27.2 version of the Karpenter helm chart?

There is a common problem when using argoCD to deploy Karpenter. There seems to be a bug in argo where old webhooks are not cleaned up. These issues are related: #3673 #1971

We are now using v0.27.5 of the Helm chart. I upgraded in the hope that this error would disappear but it didn't

I did notice that we had old webhooks a few versions ago and manually cleaned them up. But we are still seeing this error

This defaulting.webhook.karpenter.sh webhook does not exist on our cluster but the application still seems to search for it

[ellistarn]

but the application still seems to search for it

Are you running the correct image for v0.27.5? The latest version of Karpenter should not be looking for the webhook:

https://github.com/aws/karpenter-core/blob/main/pkg/webhooks/webhooks.go#L53
https://github.com/aws/karpenter-core/blob/main/pkg/webhooks/webhooks.go#L43

[seanocca]

The tag on the image was v0.27.2 not v0.27.5 (the helm chart docs had that in the values file for the release tag).
I also upgraded the CRD (I noticed I had an old version).

This fixed the issue