Karpenter VolumeUsage calculation doesn't include volumes in "attaching" state

[prad9192]

Description

Observed Behavior:

Karpenter's VolumeUsage class doesn't accurately calculate the number of volumes attached to a node, leading to potential violations of the maximum volume limit per node on AWS. It only considers currently attached volumes and ignores volumes that are in the "attaching" state. This results in all pods in Init status reporting the FailedAttachVolume error.

https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/scheduling/volumeusage.go#L187

Expected Behavior:

The VolumeUsage class should consider both attached and attaching volumes when calculating the total number of volumes on a node. This would ensure that Karpenter respects the maximum volume limit imposed by AWS and prevent pod scheduling failures

Reproduction Steps (Please include YAML):

1. Configure a Karpenter nodepool with a nitro instance type (r6).
2. Deploy a workload that requests multiple volumes > 28.
3. While the volume is in the "attaching" state, quickly deploy another workload that also requests new set of volumes.
4. Karpenter might provision the second workload on the same node, exceeding the maximum volume limit, because it doesn't consider the "attaching" volume in its calculations.
5. Sample workload to reproduce this issue is added below, try creating the same deployment in 5 different namespaces should notice the issue. Notice the screen shot below, there are currently 75 volumes attached to r6 instance.

  Warning  FailedAttachVolume  77s (x2 over 5m22s)  attachdetach-controller  (combined from similar events):
   AttachVolume.Attach failed for volume "pvc-8034a633-9bfb-4616-9527-7f9e27b47aca" : rpc error: code = Internal desc =
   Could not attach volume "vol-058bfaabdbebd0048" to node "i-0d41ceb8453849ba3": WaitForAttachmentState
   AttachVolume error, expected device but be attached but was attaching, volumeID="vol-058bfaabdbebd0048", 
   instanceID="i-0d41ceb8453849ba3", Device="/dev/xvddv", err=operation error EC2: 
   AttachVolume, https response error StatusCode: 400, RequestID: 24f30507-32b5-4f20-a3e5-ca80df15e12b, api error 
   VolumeInUse: vol-058bfaabdbebd0048 is already attached to an instance

Sample Workload for Reproducing

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv
          mountPath: /data
      volumes:
      - name: my-pv
        persistentVolumeClaim:
          claimName: my-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data  
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-1
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-1
  template:
    metadata:
      labels:
        app: my-app-1
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-1
          mountPath: /data
      volumes:
      - name: my-pv-1
        persistentVolumeClaim:
          claimName: my-pvc-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-2
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-2
  template:
    metadata:
      labels:
        app: my-app-2
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-2
          mountPath: /data
      volumes:
      - name: my-pv-2
        persistentVolumeClaim:
          claimName: my-pvc-2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-2
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-3
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-3
  template:
    metadata:
      labels:
        app: my-app-3
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-3
          mountPath: /data
      volumes:
      - name: my-pv-3
        persistentVolumeClaim:
          claimName: my-pvc-3
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-3
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-3
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-4
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-4
  template:
    metadata:
      labels:
        app: my-app-4
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-4
          mountPath: /data
      volumes:
      - name: my-pv-4
        persistentVolumeClaim:
          claimName: my-pvc-4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-4
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-4
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-4
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-5
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-5
  template:
    metadata:
      labels:
        app: my-app-5
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-5
          mountPath: /data
      volumes:
      - name: my-pv-5
        persistentVolumeClaim:
          claimName: my-pvc-5
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-5
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-5
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-5
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-6
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-6
  template:
    metadata:
      labels:
        app: my-app-6
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-6
          mountPath: /data
      volumes:
      - name: my-pv-6
        persistentVolumeClaim:
          claimName: my-pvc-6
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-6
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-6
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-6
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-7
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-7
  template:
    metadata:
      labels:
        app: my-app-7
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-7
          mountPath: /data
      volumes:
      - name: my-pv-7
        persistentVolumeClaim:
          claimName: my-pvc-7
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-7
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-7
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-7
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-8
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-8
  template:
    metadata:
      labels:
        app: my-app-8
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-8
          mountPath: /data
      volumes:
      - name: my-pv-8
        persistentVolumeClaim:
          claimName: my-pvc-8
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-8
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-8
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-8
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-9
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-9
  template:
    metadata:
      labels:
        app: my-app-9
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-9
          mountPath: /data
      volumes:
      - name: my-pv-9
        persistentVolumeClaim:
          claimName: my-pvc-9
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-9
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-9
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-9
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-10
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-10
  template:
    metadata:
      labels:
        app: my-app-10
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-10
          mountPath: /data
      volumes:
      - name: my-pv-10
        persistentVolumeClaim:
          claimName: my-pvc-10
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-10
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-10
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-10
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-11
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-11
  template:
    metadata:
      labels:
        app: my-app-11
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-11
          mountPath: /data
      volumes:
      - name: my-pv-11
        persistentVolumeClaim:
          claimName: my-pvc-11
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-11
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-11
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-11
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-12
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-12
  template:
    metadata:
      labels:
        app: my-app-12
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-12
          mountPath: /data
      volumes:
      - name: my-pv-12
        persistentVolumeClaim:
          claimName: my-pvc-12
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-12
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-12
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-12
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-13
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-13
  template:
    metadata:
      labels:
        app: my-app-13
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-13
          mountPath: /data
      volumes:
      - name: my-pv-13
        persistentVolumeClaim:
          claimName: my-pvc-13
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-13
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-13
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-13
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-14
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-14
  template:
    metadata:
      labels:
        app: my-app-14
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-14
          mountPath: /data
      volumes:
      - name: my-pv-14
        persistentVolumeClaim:
          claimName: my-pvc-14
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-14
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-14
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-14
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-15
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-15
  template:
    metadata:
      labels:
        app: my-app-15
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-15
          mountPath: /data
      volumes:
      - name: my-pv-15
        persistentVolumeClaim:
          claimName: my-pvc-15
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-15
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-15
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-15
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-16
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-16
  template:
    metadata:
      labels:
        app: my-app-16
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-16
          mountPath: /data
      volumes:
      - name: my-pv-16
        persistentVolumeClaim:
          claimName: my-pvc-16
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-16
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-16
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-16
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-17
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-17
  template:
    metadata:
      labels:
        app: my-app-17
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-17
          mountPath: /data
      volumes:
      - name: my-pv-17
        persistentVolumeClaim:
          claimName: my-pvc-17
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-17
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-17
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-17
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-18
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-18
  template:
    metadata:
      labels:
        app: my-app-18
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-18
          mountPath: /data
      volumes:
      - name: my-pv-18
        persistentVolumeClaim:
          claimName: my-pvc-18
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-18
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-18
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-18
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-19
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-19
  template:
    metadata:
      labels:
        app: my-app-19
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-19
          mountPath: /data
      volumes:
      - name: my-pv-19
        persistentVolumeClaim:
          claimName: my-pvc-19
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-19
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-19
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-19
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-20
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-20
  template:
    metadata:
      labels:
        app: my-app-20
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-20
          mountPath: /data
      volumes:
      - name: my-pv-20
        persistentVolumeClaim:
          claimName: my-pvc-20
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-20
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-20
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-20
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-21
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-21
  template:
    metadata:
      labels:
        app: my-app-21
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-21
          mountPath: /data
      volumes:
      - name: my-pv-21
        persistentVolumeClaim:
          claimName: my-pvc-21
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-21
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-21
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-21
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-22
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-22
  template:
    metadata:
      labels:
        app: my-app-22
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-22
          mountPath: /data
      volumes:
      - name: my-pv-22
        persistentVolumeClaim:
          claimName: my-pvc-22
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-22
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-22
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-22
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-23
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-23
  template:
    metadata:
      labels:
        app: my-app-23
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-23
          mountPath: /data
      volumes:
      - name: my-pv-23
        persistentVolumeClaim:
          claimName: my-pvc-23
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-23
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-23
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-23
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-24
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-24
  template:
    metadata:
      labels:
        app: my-app-24
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-24
          mountPath: /data
      volumes:
      - name: my-pv-24
        persistentVolumeClaim:
          claimName: my-pvc-24
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-24
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-24
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-24
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-25
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-25
  template:
    metadata:
      labels:
        app: my-app-25
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-25
          mountPath: /data
      volumes:
      - name: my-pv-25
        persistentVolumeClaim:
          claimName: my-pvc-25
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-25
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-25
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-25
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-26
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-26
  template:
    metadata:
      labels:
        app: my-app-26
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-26
          mountPath: /data
      volumes:
      - name: my-pv-26
        persistentVolumeClaim:
          claimName: my-pvc-26
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-26
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-26
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-26
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-27
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-27
  template:
    metadata:
      labels:
        app: my-app-27
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-27
          mountPath: /data
      volumes:
      - name: my-pv-27
        persistentVolumeClaim:
          claimName: my-pvc-27
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-27
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-27
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-27
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-28
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-28
  template:
    metadata:
      labels:
        app: my-app-28
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-28
          mountPath: /data
      volumes:
      - name: my-pv-28
        persistentVolumeClaim:
          claimName: my-pvc-28
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-28
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-28
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-28
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-29
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-29
  template:
    metadata:
      labels:
        app: my-app-29
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-29
          mountPath: /data
      volumes:
      - name: my-pv-29
        persistentVolumeClaim:
          claimName: my-pvc-29
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-29
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-29
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-29
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pv-deployment-30
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: my-app-30
  template:
    metadata:
      labels:
        app: my-app-30
    spec:
      nodeSelector:
        affinity: tekton
      tolerations:
      - effect: NoSchedule
        key: workerrole
        operator: Equal
        value: tekton
      containers:
      - name: my-container
        image: nginx:latest
        volumeMounts:
        - name: my-pv-30
          mountPath: /data
      volumes:
      - name: my-pv-30
        persistentVolumeClaim:
          claimName: my-pvc-30
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-30
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-30
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: /data-30

Versions:

• Chart Version: v0.35.5
• Kubernetes version (use kubectl version):
  Client Version: v1.31.2
  Kustomize Version: v5.4.2
  Server Version: v1.29.10-eks-7f9249a

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[prad9192]

Ah, the issue was the missing startupTaint.

Argocd was resetting the startupTaint on the nodepool.

Fix- #260 (comment)