"error":"Internal error occurred: failed calling webhook \"validation.webhook.karpenter.sh\"

[dhanutalari]

Description

I attempted to upgrade Karpenter from version v0.37.0 to v1.0.1. Following the documentation, I first upgraded to the minor version v0.37.2, enabling webhooks and patching NodePools, NodeClaims, and EC2NodeClass, which worked fine.

Later, I upgraded to version v1.0.1. However, after the upgrade, the Karpenter controller did not function as expected, and I encountered the following error:
"{"level":"ERROR","time":"2024-08-27T07:49:42.502Z","logger":"controller","message":"Reconciler error","commit":"62a726c","controller":"nodeclaim.tagging","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"nats-nmjlx"},"namespace":"","name":"nats-nmjlx","reconcileID":"efb1972f-570f-4bad-b952-130449b299c7","error":"Internal error occurred: failed calling webhook "validation.webhook.karpenter.sh": failed to call webhook: the server rejected our request for an unknown reason"}"

I applied the NodePool, NodeClass, and EC2NodeClass CRDs using the following commands:

kubectl apply -f
"https://raw.githubusercontent.com/aws/karpenter-provider-aws/v1.0.1/pkg/apis/crds/karpenter.sh_nodepools.yaml"
kubectl apply -f
"https://raw.githubusercontent.com/aws/karpenter-provider-aws/v1.0.1/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml"
kubectl apply -f
"https://raw.githubusercontent.com/aws/karpenter-provider-aws/v1.0.1/pkg/apis/crds/karpenter.sh_nodeclaims.yaml"

I then generated and applied the Karpenter YAML file as follows:

helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version "1.0.1" --namespace "kube-system"
--set "settings.clusterName=nonprod-eks"
--set "settings.interruptionQueue=nonprod-eks"
--set "serviceAccount.annotations.eks.amazonaws.com/role-arn=arn:aws:iam::852911611684:role/KarpenterControllerRole-nonprod-eks"
--set controller.resources.requests.cpu=1
--set controller.resources.requests.memory=1Gi
--set controller.resources.limits.cpu=1
--set controller.resources.limits.memory=1Gi > karpenter.yaml

Please advise on how to resolve the this issue .

[rschalo]

Could you please share what your current Karpenter deployment looks like?

[dhanutalari]

@rschalo find version : v0.37.2 deployment file and v1.0.1 deployment files below

version: v0.37.2

Source: karpenter/templates/poddisruptionbudget.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter

Source: karpenter/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::852911611684:role/KarpenterControllerRole-nonprod-eks

Source: karpenter/templates/secret-webhook-cert.yaml

apiVersion: v1
kind: Secret
metadata:
name: karpenter-cert
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm

data: {} # Injected by karpenter-webhook

---

Source: karpenter/templates/aggregate-clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter-admin
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]
• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]

---

Source: karpenter/templates/clusterrole-core.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter-core
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"]
  verbs: ["get", "list", "watch"]
• apiGroups: [""]
  resources: ["pods", "nodes", "persistentvolumes", "persistentvolumeclaims", "replicationcontrollers", "namespaces"]
  verbs: ["get", "list", "watch"]
• apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses", "csinodes"]
  verbs: ["get", "watch", "list"]
• apiGroups: ["apps"]
  resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
  verbs: ["list", "watch"]
• apiGroups: ["admissionregistration.k8s.io"]
  resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
  verbs: ["get", "watch", "list"]
• apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["watch", "list"]
• apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["karpenter.sh"]
  resources: ["nodeclaims", "nodeclaims/status"]
  verbs: ["create", "delete", "update", "patch"]
• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status"]
  verbs: ["update", "patch"]
• apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
• apiGroups: [""]
  resources: ["nodes"]
  verbs: ["patch", "delete"]
• apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
• apiGroups: ["admissionregistration.k8s.io"]
  resources: ["validatingwebhookconfigurations"]
  verbs: ["update"]
  resourceNames: ["validation.webhook.karpenter.sh", "validation.webhook.config.karpenter.sh"]
• apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["update"]

---

Source: karpenter/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses", "ec2nodeclasses/status"]
  verbs: ["patch", "update"]
• apiGroups: ["admissionregistration.k8s.io"]
  resources: ["validatingwebhookconfigurations"]
  verbs: ["update"]
  resourceNames: ["validation.webhook.karpenter.k8s.aws"]
• apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations"]
  verbs: ["update"]
  resourceNames: ["defaulting.webhook.karpenter.k8s.aws"]

---

Source: karpenter/templates/clusterrole-core.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: karpenter-core
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: karpenter-core
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: karpenter
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: karpenter
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch"]
• apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: [""]
  resources: ["secrets"]
  verbs: ["update"]
  resourceNames:
  • "karpenter-cert"
• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["patch", "update"]
  resourceNames:
  • "karpenter-leader-election"

Cannot specify resourceNames on create

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create"]

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter-dns
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: [""]
  resources: ["services"]
  resourceNames: ["kube-dns"]
  verbs: ["get"]

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter-lease
namespace: kube-node-lease
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["delete"]

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter-dns
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter-dns
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter-lease
namespace: kube-node-lease
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter-lease
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- name: http-metrics
port: 8000
targetPort: http-metrics
protocol: TCP
- name: webhook-metrics
port: 8001
targetPort: webhook-metrics
protocol: TCP
- name: https-webhook
port: 8443
targetPort: https-webhook
protocol: TCP
selector:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter

Source: karpenter/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 2
revisionHistoryLimit: 10
strategy:
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
template:
metadata:
labels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
annotations:
spec:
serviceAccountName: karpenter
securityContext:
fsGroup: 65532
priorityClassName: "system-cluster-critical"
dnsPolicy: ClusterFirst
containers:
- name: karpenter-filebeat
image: 852911611684.dkr.ecr.ap-south-1.amazonaws.com/nonprod-filebeat:latest
volumeMounts:
- name: karpoenter-logs
mountPath: /usr/local/karpenter
- name: filebeat-config
mountPath: /usr/share/filebeat/filebeat.yml
subPath: filebeat.yml
- name: controller
securityContext:
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
image: public.ecr.aws/karpenter/controller:0.37.2@sha256:0402d38370aca70cc976b1f9b64fc3c50c88c8fe281dc39d0300df89a62bd16e
imagePullPolicy: IfNotPresent
env:
- name: KUBERNETES_MIN_VERSION
value: "1.19.0-0"
- name: KARPENTER_SERVICE
value: karpenter
- name: WEBHOOK_PORT
value: "8443"
- name: WEBHOOK_METRICS_PORT
value: "8001"
- name: DISABLE_WEBHOOK
value: "false"
- name: LOG_LEVEL
value: "info"
- name: METRICS_PORT
value: "8000"
- name: HEALTH_PROBE_PORT
value: "8081"
- name: SYSTEM_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: controller
divisor: "0"
resource: limits.memory
- name: FEATURE_GATES
value: "Drift=true,SpotToSpotConsolidation=true"
- name: BATCH_MAX_DURATION
value: "10s"
- name: BATCH_IDLE_DURATION
value: "1s"
- name: ASSUME_ROLE_DURATION
value: "15m"
- name: CLUSTER_NAME
value: "nonprod-eks"
- name: VM_MEMORY_OVERHEAD_PERCENT
value: "0.075"
- name: INTERRUPTION_QUEUE
value: "nonprod-eks"
- name: RESERVED_ENIS
value: "0"
ports:
- name: http-metrics
containerPort: 8000
protocol: TCP
- name: webhook-metrics
containerPort: 8001
protocol: TCP
- name: https-webhook
containerPort: 8443
protocol: TCP
- name: http
containerPort: 8081
protocol: TCP
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 30
httpGet:
path: /healthz
port: http
readinessProbe:
initialDelaySeconds: 5
timeoutSeconds: 30
httpGet:
path: /readyz
port: http
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
nodeSelector:
kubernetes.io/os: linux
# The template below patches the .Values.affinity to add a default label selector where not specificed
volumes:
- name: karpoenter-logs
hostPath:
path: /var/log/pods
- name: filebeat-config
configMap:
name: filebeat-configmap
items:
- key: filebeat.yml
path: filebeat.yml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
- key: eks.amazonaws.com/nodegroup
operator: In
values:
- karpenter-nodegroup
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
topologyKey: kubernetes.io/hostname
# The template below patches the .Values.topologySpreadConstraints to add a default label selector where not specificed
topologySpreadConstraints:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: "app"
value: "karpenter"
operator: "Equal"
effect: "NoSchedule"

Source: karpenter/templates/webhooks.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: defaulting.webhook.karpenter.k8s.aws
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
webhooks:

• name: defaulting.webhook.karpenter.k8s.aws
  admissionReviewVersions: ["v1"]
  clientConfig:
  service:
  name: karpenter
  namespace: kube-system
  port: 8443
  failurePolicy: Fail
  sideEffects: None
  rules:
  • apiGroups:
    • karpenter.k8s.aws
      apiVersions:
    • v1beta1
      operations:
    • CREATE
    • UPDATE
      resources:
    • ec2nodeclasses
    • ec2nodeclasses/status
      scope: '*'

---

Source: karpenter/templates/webhooks-core.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: validation.webhook.karpenter.sh
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
webhooks:

• name: validation.webhook.karpenter.sh
  admissionReviewVersions: ["v1"]
  clientConfig:
  service:
  name: karpenter
  namespace: kube-system
  port: 8443
  failurePolicy: Fail
  sideEffects: None
  rules:
  • apiGroups:
    • karpenter.sh
      apiVersions:
    • v1beta1
      operations:
    • CREATE
    • UPDATE
      resources:
    • nodeclaims
    • nodeclaims/status
      scope: '*'
  • apiGroups:
    • karpenter.sh
      apiVersions:
    • v1beta1
      operations:
    • CREATE
    • UPDATE
      resources:
    • nodepools
    • nodepools/status
      scope: '*'

---

Source: karpenter/templates/webhooks-core.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: validation.webhook.config.karpenter.sh
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
webhooks:

• name: validation.webhook.config.karpenter.sh
  admissionReviewVersions: ["v1"]
  clientConfig:
  service:
  name: karpenter
  namespace: kube-system
  port: 8443
  failurePolicy: Fail
  sideEffects: None
  objectSelector:
  matchLabels:
  app.kubernetes.io/part-of: karpenter

---

Source: karpenter/templates/webhooks.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: validation.webhook.karpenter.k8s.aws
labels:
helm.sh/chart: karpenter-0.37.2
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "0.37.2"
app.kubernetes.io/managed-by: Helm
webhooks:

• name: validation.webhook.karpenter.k8s.aws
  admissionReviewVersions: ["v1"]
  clientConfig:
  service:
  name: karpenter
  namespace: kube-system
  port: 8443
  failurePolicy: Fail
  sideEffects: None
  rules:
  • apiGroups:
    • karpenter.k8s.aws
      apiVersions:
    • v1beta1
      operations:
    • CREATE
    • UPDATE
      resources:
    • ec2nodeclasses
    • ec2nodeclasses/status
      scope: '*'

for version : v1.0.1

---

Source: karpenter/templates/poddisruptionbudget.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter

Source: karpenter/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::852911611684:role/KarpenterControllerRole-nonprod-eks

Source: karpenter/templates/secret-webhook-cert.yaml

apiVersion: v1
kind: Secret
metadata:
name: karpenter-cert
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm

data: {} # Injected by karpenter-webhook

---

Source: karpenter/templates/aggregate-clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter-admin
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]
• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]

---

Source: karpenter/templates/clusterrole-core.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter-core
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"]
  verbs: ["get", "list", "watch"]
• apiGroups: [""]
  resources: ["pods", "nodes", "persistentvolumes", "persistentvolumeclaims", "replicationcontrollers", "namespaces"]
  verbs: ["get", "list", "watch"]
• apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses", "csinodes", "volumeattachments"]
  verbs: ["get", "watch", "list"]
• apiGroups: ["apps"]
  resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
  verbs: ["list", "watch"]
• apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["watch", "list"]
• apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["karpenter.sh"]
  resources: ["nodeclaims", "nodeclaims/status"]
  verbs: ["create", "delete", "update", "patch"]
• apiGroups: ["karpenter.sh"]
  resources: ["nodepools", "nodepools/status"]
  verbs: ["update", "patch"]
• apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
• apiGroups: [""]
  resources: ["nodes"]
  verbs: ["patch", "delete", "update"]
• apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
• apiGroups: [""]
  resources: ["pods"]
  verbs: ["delete"]
• apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["update"]

---

Source: karpenter/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: karpenter
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["karpenter.k8s.aws"]
  resources: ["ec2nodeclasses", "ec2nodeclasses/status"]
  verbs: ["patch", "update"]

---

Source: karpenter/templates/clusterrole-core.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: karpenter-core
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: karpenter-core
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: karpenter
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: karpenter
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch"]
• apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: [""]
  resources: ["secrets"]
  verbs: ["update"]
  resourceNames:
  • "karpenter-cert"
• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["patch", "update"]
  resourceNames:
  • "karpenter-leader-election"

Cannot specify resourceNames on create

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create"]

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter-dns
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: [""]
  resources: ["services"]
  resourceNames: ["kube-dns"]
  verbs: ["get"]

---

Source: karpenter/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: karpenter-lease
namespace: kube-node-lease
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
rules:

Read

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "list", "watch"]

Write

• apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["delete"]

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter-dns
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter-dns
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: karpenter-lease
namespace: kube-node-lease
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: karpenter-lease
subjects:

• kind: ServiceAccount
  name: karpenter
  namespace: kube-system

---

Source: karpenter/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- name: http-metrics
port: 8080
targetPort: http-metrics
protocol: TCP
- name: webhook-metrics
port: 8001
targetPort: webhook-metrics
protocol: TCP
- name: https-webhook
port: 8443
targetPort: https-webhook
protocol: TCP
selector:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter

Source: karpenter/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: karpenter
namespace: kube-system
labels:
helm.sh/chart: karpenter-1.0.1
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.1"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 2
revisionHistoryLimit: 10
strategy:
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
template:
metadata:
labels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
annotations:
spec:
serviceAccountName: karpenter
securityContext:
fsGroup: 65532
priorityClassName: "system-cluster-critical"
dnsPolicy: ClusterFirst
containers:
- name: karpenter-filebeat
image: 852911611684.dkr.ecr.ap-south-1.amazonaws.com/nonprod-filebeat:latest
volumeMounts:
- name: karpoenter-logs
mountPath: /usr/local/karpenter
- name: filebeat-config
mountPath: /usr/share/filebeat/filebeat.yml
subPath: filebeat.yml
- name: controller
securityContext:
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
image: public.ecr.aws/karpenter/controller:1.0.1@sha256:fc54495b35dfeac6459ead173dd8452ca5d572d90e559f09536a494d2795abe6
imagePullPolicy: IfNotPresent
env:
- name: KUBERNETES_MIN_VERSION
value: "1.19.0-0"
- name: KARPENTER_SERVICE
value: karpenter
- name: WEBHOOK_PORT
value: "8443"
- name: WEBHOOK_METRICS_PORT
value: "8001"
- name: DISABLE_WEBHOOK
value: "false"
- name: LOG_LEVEL
value: "info"
- name: METRICS_PORT
value: "8080"
- name: HEALTH_PROBE_PORT
value: "8081"
- name: SYSTEM_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: controller
divisor: "0"
resource: limits.memory
- name: FEATURE_GATES
value: "SpotToSpotConsolidation=true"
- name: BATCH_MAX_DURATION
value: "10s"
- name: BATCH_IDLE_DURATION
value: "1s"
- name: CLUSTER_NAME
value: "nonprod-eks"
- name: VM_MEMORY_OVERHEAD_PERCENT
value: "0.075"
- name: INTERRUPTION_QUEUE
value: "nonprod-eks"
- name: RESERVED_ENIS
value: "0"
ports:
- name: http-metrics
containerPort: 8080
protocol: TCP
- name: webhook-metrics
containerPort: 8001
protocol: TCP
- name: https-webhook
containerPort: 8443
protocol: TCP
- name: http
containerPort: 8081
protocol: TCP
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 30
httpGet:
path: /healthz
port: http
readinessProbe:
initialDelaySeconds: 5
timeoutSeconds: 30
httpGet:
path: /readyz
port: http
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
nodeSelector:
kubernetes.io/os: linux
# The template below patches the .Values.affinity to add a default label selector where not specificed
volumes:
- name: karpoenter-logs
hostPath:
path: /var/log/pods
- name: filebeat-config
configMap:
name: filebeat-configmap
items:
- key: filebeat.yml
path: filebeat.yml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
- key: eks.amazonaws.com/nodegroup
operator: In
values:
- karpenter-nodegroup
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
topologyKey: kubernetes.io/hostname
# The template below patches the .Values.topologySpreadConstraints to add a default label selector where not specificed
topologySpreadConstraints:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: "app"
value: "karpenter"
operator: "Equal"
effect: "NoSchedule"

i just added custome nodegroup for karpenter to place and filebeat container to push karpenter controller logs to elasticsearch .

[gnadaban]

It's super difficult to interpret this unstructured datadump, can you instead create and link a public gist with a single YAML file?

[gnadaban]

I think it's not spelled out in the upgrade documentation, but apparently the issue goes away if you delete the Validating and Mutating webhook registrations that were created by Karpenter 0.37 after upgrading to 1.0

[dhanutalari]

issue fixed thanks @gnadaban @rschalo .

[booleanbetrayal]

Thank you for this @gnadaban ! This upgrade has been .... quite the process.