Support region topology for the ebs in tree provider #1109

Noksa · 2022-01-08T05:58:40Z

Version

Karpenter: v0.5.4

Kubernetes: v1.21.5

Expected Behavior

Karpenter creates nodes using volume topology when a PVC already exists for a pod

Actual Behavior

Karpenter can't create a new node due to wrong nodeSelector

Steps to Reproduce the Problem

Use the following steps and manifests.

Apply manifests first time (a node will be provisioned)
Scale STS to 0
Wait until the node deleted
Scale STS to 1
Get the error

So it looks like Karpenter can't provision nodes if a PVC already exists for a pod.

Manifests:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
    - port: 80
      name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        only-db: "true"
      tolerations:
        - key: "only-db"
          operator: Exists
          effect: NoSchedule
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi

Resource Specs and Logs

Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  annotations:
    meta.helm.sh/release-name: smb-eu-central-1
    meta.helm.sh/release-namespace: smb-eu-central-1
  creationTimestamp: "2022-01-08T05:39:28Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: storage-db-smb-eu-central-1
  resourceVersion: "171015299"
  uid: 3e707cf2-a6a0-4b06-b8ae-5d8675722cbc
spec:
  kubeletConfiguration: {}
  labels:
    only-db: "true"
  limits: {}
  provider:
    apiVersion: extensions.karpenter.sh/v1alpha1
    instanceProfile: ad-eks-c-NodeInstanceProfile
    kind: AWS
    launchTemplate: ad-eks-db-lt
    securityGroupSelector:
      aws:eks:cluster-name: ad-eks-c
    subnetSelector:
      kubernetes.io/cluster/ad-eks-c: '*'
  requirements:
  - key: kubernetes.io/arch
    operator: In
    values:
    - amd64
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - on-demand
  taints:
  - effect: NoSchedule
    key: only-db
    value: "true"
  ttlSecondsAfterEmpty: 120
status:
  resources:
    cpu: "0"
    memory: "0"

Log:

tried provisioner/storage-db-smb-eu-central-1: invalid nodeSelector "topology.kubernetes.io/region", [eu-central-1] not in []	{"commit": "7e79a67", "pod": "kazoo-db/storage-db-smb-eu-central-1-0"}

Part of pod spec:

      nodeSelector:
        only-db: "true"
      tolerations:
      - effect: NoSchedule
        key: only-db
        value: "true"
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app: db
            release: smb-eu-central-1
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
      - labelSelector:
          matchLabels:
            app: db
            release: smb-eu-central-1
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule

As I can see Karpenter tries to use some other nodeSelector for some reason.

The text was updated successfully, but these errors were encountered:

ellistarn · 2022-01-08T07:41:03Z

It looks like the topology region label is getting applied to your volume somehow. Is it possible that this label is a property on your storage class for some reason?

Karpenter doesn't support (currently) the region label since it doesn't support multiple regions. We could always add noop support for it, but I'm curious how this is getting applied in the first place.

Noksa · 2022-01-08T08:11:50Z

Yeah, my volumes have the following labels:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: aws-ebs-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
  creationTimestamp: "2021-12-27T08:10:47Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    topology.kubernetes.io/region: eu-central-1
    topology.kubernetes.io/zone: eu-central-1a
...

And the following nodeAffinity:

...
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - eu-central-1a
        - key: topology.kubernetes.io/region
          operator: In
          values:
          - eu-central-1
...

But the StorageClass doesn't have any unexpected settings:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2021-01-26T12:45:33Z"
  name: gp2
  resourceVersion: "76963236"
  uid: d2acc111-2227-4953-9c5d-ea09084b70a0
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

I have 3 EKS clusters in different regions and all of them have the same behavior.

I see the difference that I'm using default gp2 provisioner kubernetes.io/aws-ebs.
But in #1015 you used ebs csi

olemarkus · 2022-01-08T08:32:57Z

I know of a couple of configurations that will cause this. I don't know what EKS use, but I am guessing it depends on versions.

First off, there is this KCM admission controller
Secondly, it is CCMs GetLabelsForVolume. I believe this is only used if you are using the in-tree EBS volume driver, which you are (the kubernetes.io/aws-ebs bit).
If you are using the AWS EBS CSI driver, I believe these labels would not be set.

Noksa · 2022-01-08T08:45:13Z

Yeah, I've tested it.

When I use https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore
The labels and the nodeAffinity are set to volumes.

When I use AWS EBS CSI driver the volume looks like this and seems it will work as expected:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
  creationTimestamp: "2022-01-08T08:41:35Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/ebs-csi-aws-com
  name: pvc-2986b2aa-d788-4549-a9b7-01bb8198dc48
  resourceVersion: "171094109"
  uid: f91cbd91-a115-4c3e-b39e-77d5dc6b62f4
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 4Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: ebs-claim
    namespace: smb-eu-central-1
    resourceVersion: "171093369"
    uid: 2986b2aa-d788-4549-a9b7-01bb8198dc48
  csi:
    driver: ebs.csi.aws.com
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: ...
    volumeHandle: ...
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.ebs.csi.aws.com/zone
          operator: In
          values:
          - eu-central-1a
  persistentVolumeReclaimPolicy: Delete
  storageClassName: ebs-sc
  volumeMode: Filesystem
status:
  phase: Bound

At the same time https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore is default EBS provisioner in EKS clusters at the moment

ellistarn · 2022-01-08T17:03:45Z

How difficult is it for you to migrate to the out-of-tree provider? I think we likely want to support the in-tree in the short term, but perhaps not in the long term.

Noksa · 2022-01-08T17:37:13Z

@ellistarn
I already have moved to the ebs csi driver due to the issue.
So it is not a problem anymore. At least for me.

This issue was a good point to try an out-of-tree storage provider (:

Noksa · 2022-01-08T17:40:39Z

Probably it would be good to update the documentation about this limitation to avoid new similar issues

jeffreymlewis · 2022-01-11T21:56:31Z

I ran into this issue yesterday. Having support for the in-tree provisioner would be greatly appreciated! I will look into switching to the AWS EBS CSI driver, but I assume that will be too much churn right now.

DWSR · 2022-01-19T19:40:53Z

We're running into an issue where the in-tree provisioner used the deprecated failure-domain.beta.kubernetes.io/{region,zone} labels, and so Karpenter will not select it:

2022-01-19T19:37:11.206Z	DEBUG	controller.selection	Could not schedule pod, matched 0/1 provisioners, tried provisioner/default: invalid nodeSelector "failure-domain.beta.kubernetes.io/region", [us-east-2] not in []	{"commit": "7e79a67", "pod": "cockroachdb-system/cockroachdb-1"}

It would be really helpful if Karpenter could support these labels for those of us still using in-tree provisioned volumes.

More context: https://gist.github.com/DWSR/ee176d7cb1d7678ebbe23f8b38ae2885

tobernguyen · 2022-01-20T00:27:16Z

We're running into this same issue that prevents us from adopting Karpenter unfortunately. A workaround to support in-tree provisioner would be greatly appreciated because migrating in-tree PV to CSI PV won't be supported in EKS in foreseeable future: kubernetes-sigs/aws-ebs-csi-driver#480 (comment)

vietwow · 2022-01-26T18:10:38Z

I also got the same issue. Really wait for this feature to be supported soon. Thanks

lbvffvbl · 2022-02-01T20:05:34Z

hi, Folks. Looks like you should add v1.LabelFailureDomainBetaRegion to IgnoredLabels too
because v1.LabelTopologyRegion - it's "topology.kubernetes.io/region" and i got error

invalid nodeSelector "failure-domain.beta.kubernetes.io/region", [us-east-2] not in []

until add v1.LabelFailureDomainBetaRegion (failure-domain.beta.kubernetes.io/region) to IgnoredLabels

tobernguyen · 2022-02-03T21:41:30Z

hi, Folks. Looks like you should add v1.LabelFailureDomainBetaRegion to IgnoredLabels too because v1.LabelTopologyRegion - it's "topology.kubernetes.io/region" and i got error
invalid nodeSelector "failure-domain.beta.kubernetes.io/region", [us-east-2] not in []
until add v1.LabelFailureDomainBetaRegion (failure-domain.beta.kubernetes.io/region) to IgnoredLabels

I'm using v0.6.0 and have the exact same issue.

ellistarn · 2022-02-03T21:59:00Z

https://github.com/aws/karpenter/pull/1271/files

Noksa added the bug Something isn't working label Jan 8, 2022

Noksa changed the title ~~Karpenter doesn't provision pods with persistent volumes (StatefulSet)~~ Karpenter doesn't provision nodes when PVC already exist for a pod (StatefulSet) Jan 8, 2022

Noksa changed the title ~~Karpenter doesn't provision nodes when PVC already exist for a pod (StatefulSet)~~ Karpenter doesn't provision nodes when a PVC already exists for a pod (StatefulSet) Jan 8, 2022

ellistarn changed the title ~~Karpenter doesn't provision nodes when a PVC already exists for a pod (StatefulSet)~~ Support region topology for the ebs in tree provider Jan 8, 2022

ellistarn added documentation Improvements or additions to documentation feature New feature or request and removed bug Something isn't working labels Jan 8, 2022

bwagner5 mentioned this issue Jan 25, 2022

[Question] Karpenter doesn't scale out #1218

Closed

bwagner5 mentioned this issue Jan 26, 2022

support region wellknown label #1228

Merged

3 tasks

bwagner5 closed this as completed Jan 31, 2022

ellistarn mentioned this issue Feb 3, 2022

Added failure-domain.beta.kubernetes.io/region to NormalizedLabels #1271

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support region topology for the ebs in tree provider #1109

Support region topology for the ebs in tree provider #1109

Noksa commented Jan 8, 2022 •

edited

Loading

ellistarn commented Jan 8, 2022

Noksa commented Jan 8, 2022 •

edited

Loading

olemarkus commented Jan 8, 2022

Noksa commented Jan 8, 2022 •

edited

Loading

ellistarn commented Jan 8, 2022

Noksa commented Jan 8, 2022

Noksa commented Jan 8, 2022 •

edited

Loading

jeffreymlewis commented Jan 11, 2022 •

edited

Loading

DWSR commented Jan 19, 2022 •

edited

Loading

tobernguyen commented Jan 20, 2022 •

edited

Loading

vietwow commented Jan 26, 2022

lbvffvbl commented Feb 1, 2022 •

edited

Loading

tobernguyen commented Feb 3, 2022

ellistarn commented Feb 3, 2022

Support region topology for the ebs in tree provider #1109

Support region topology for the ebs in tree provider #1109

Comments

Noksa commented Jan 8, 2022 • edited Loading

Version

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Resource Specs and Logs

ellistarn commented Jan 8, 2022

Noksa commented Jan 8, 2022 • edited Loading

olemarkus commented Jan 8, 2022

Noksa commented Jan 8, 2022 • edited Loading

ellistarn commented Jan 8, 2022

Noksa commented Jan 8, 2022

Noksa commented Jan 8, 2022 • edited Loading

jeffreymlewis commented Jan 11, 2022 • edited Loading

DWSR commented Jan 19, 2022 • edited Loading

tobernguyen commented Jan 20, 2022 • edited Loading

vietwow commented Jan 26, 2022

lbvffvbl commented Feb 1, 2022 • edited Loading

tobernguyen commented Feb 3, 2022

ellistarn commented Feb 3, 2022

Noksa commented Jan 8, 2022 •

edited

Loading

Noksa commented Jan 8, 2022 •

edited

Loading

Noksa commented Jan 8, 2022 •

edited

Loading

Noksa commented Jan 8, 2022 •

edited

Loading

jeffreymlewis commented Jan 11, 2022 •

edited

Loading

DWSR commented Jan 19, 2022 •

edited

Loading

tobernguyen commented Jan 20, 2022 •

edited

Loading

lbvffvbl commented Feb 1, 2022 •

edited

Loading