Pod with Volume couldn't be scheduled on any node

[sylr]

Version

Karpenter: v0.10.1

Kubernetes: v1.22.6-eks-7d68063

Expected Behavior

Actual Behavior

I tried to upgrade a redis cluster deployed with bitnami's chart and karpenter failed to schedule the new redis-master pod (which is managed by a stateful set and has a PVC volume).

The pod remain in pending state:

  Warning  FailedScheduling        7m43s                  default-scheduler        0/8 nodes are available: 1 Insufficient memory, 1 node(s) had volume node affinity conflict, 2 node(s) didn't match pod topology spread constraints, 2 node(s) had taint {company.com/dedicated: ingress}, that the pod didn't tolerate, 2 node(s) had taint {company.com/dedicated: monitoring}, that the pod didn't tolerate.
  Warning  FailedScheduling        105s (x13 over 9m27s)  default-scheduler        0/8 nodes are available: 2 Insufficient memory, 2 node(s) didn't match pod topology spread constraints, 2 node(s) had taint {company.com/dedicated: ingress}, that the pod didn't tolerate, 2 node(s) had taint {company.com/dedicated: monitoring}, that the pod didn't tolerate.

Here are karpenters logs with debug level:

2022-06-02T12:13:05.234Z	DEBUG	controller	Discovered subnets: [subnet-097dd9e57d1555d6d (eu-central-1c) subnet-09b3a2ff69674c2c6 (eu-central-1b) subnet-0ed948e68264d80b2 (eu-central-1a)]	{"commit": "10fc37b"}
2022-06-02T12:13:05.236Z	INFO	controller	0 pod(s) will schedule against new capacity, 1 pod(s) against existing capacity	{"commit": "10fc37b"}
2022-06-02T12:13:05.236Z	INFO	controller	Waiting for unschedulable pods	{"commit": "10fc37b"}
2022-06-02T12:13:10.170Z	INFO	controller	Batched 1 pod(s) in 1.000129842s	{"commit": "10fc37b"}
2022-06-02T12:13:10.175Z	INFO	controller	0 pod(s) will schedule against new capacity, 1 pod(s) against existing capacity	{"commit": "10fc37b"}
2022-06-02T12:13:10.175Z	INFO	controller	Waiting for unschedulable pods	{"commit": "10fc37b"}
2022-06-02T12:13:15.172Z	INFO	controller	Batched 1 pod(s) in 1.001128031s	{"commit": "10fc37b"}
2022-06-02T12:13:15.176Z	INFO	controller	0 pod(s) will schedule against new capacity, 1 pod(s) against existing capacity	{"commit": "10fc37b"}
2022-06-02T12:13:15.176Z	INFO	controller	Waiting for unschedulable pods	{"commit": "10fc37b"}
2022-06-02T12:13:20.173Z	INFO	controller	Batched 1 pod(s) in 1.00070326s	{"commit": "10fc37b"}
2022-06-02T12:13:20.176Z	INFO	controller	0 pod(s) will schedule against new capacity, 1 pod(s) against existing capacity	{"commit": "10fc37b"}
2022-06-02T12:13:20.176Z	INFO	controller	Waiting for unschedulable pods	{"commit": "10fc37b"}
2022-06-02T12:13:25.173Z	INFO	controller	Batched 1 pod(s) in 1.000849163s	{"commit": "10fc37b"}
2022-06-02T12:13:25.177Z	INFO	controller	0 pod(s) will schedule against new capacity, 1 pod(s) against existing capacity	{"commit": "10fc37b"}
2022-06-02T12:13:25.177Z	INFO	controller	Waiting for unschedulable pods	{"commit": "10fc37b"}

I managed to solve the problem by draining an existing node where I knew the pod could be scheduled and uncordonned it right away:

2022-06-02T12:14:53.286Z        INFO    controller      Batched 13 pod(s) in 2.77251638s        {"commit": "10fc37b"}
2022-06-02T12:14:53.291Z        INFO    controller      0 pod(s) will schedule against new capacity, 13 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:14:53.291Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:14:55.186Z        INFO    controller      Batched 13 pod(s) in 1.001030003s       {"commit": "10fc37b"}
2022-06-02T12:14:55.205Z        INFO    controller      0 pod(s) will schedule against new capacity, 13 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:14:55.205Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:14:57.480Z        INFO    controller      Batched 13 pod(s) in 1.96590421s        {"commit": "10fc37b"}
2022-06-02T12:14:57.486Z        INFO    controller      0 pod(s) will schedule against new capacity, 13 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:14:57.486Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:15:02.481Z        INFO    controller      Batched 15 pod(s) in 3.29655108s        {"commit": "10fc37b"}
2022-06-02T12:15:02.487Z        INFO    controller      0 pod(s) will schedule against new capacity, 15 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:15:02.487Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:15:07.482Z        INFO    controller      Batched 15 pod(s) in 3.29704561s        {"commit": "10fc37b"}
2022-06-02T12:15:07.488Z        INFO    controller      0 pod(s) will schedule against new capacity, 15 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:15:07.488Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:15:10.186Z        INFO    controller      Batched 13 pod(s) in 1.000029751s       {"commit": "10fc37b"}
2022-06-02T12:15:10.191Z        INFO    controller      0 pod(s) will schedule against new capacity, 13 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:15:10.191Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:15:10.367Z        INFO    controller.node Added TTL to empty node {"commit": "10fc37b", "node": "ip-10-132-25-58.eu-central-1.compute.internal"}
2022-06-02T12:15:10.402Z        INFO    controller.node Added TTL to empty node {"commit": "10fc37b", "node": "ip-10-132-25-58.eu-central-1.compute.internal"}
2022-06-02T12:15:12.482Z        INFO    controller      Batched 13 pod(s) in 1.966546141s       {"commit": "10fc37b"}
2022-06-02T12:15:12.590Z        DEBUG   controller      Discovered subnets: [subnet-097dd9e57d1555d6d (eu-central-1c) subnet-09b3a2ff69674c2c6 (eu-central-1b) subnet-0ed948e68264d80b2 (eu-central-1a)]        {"commit": "10fc37b"}
2022-06-02T12:15:12.594Z        INFO    controller      0 pod(s) will schedule against new capacity, 13 pod(s) against existing capacity        {"commit": "10fc37b"}
2022-06-02T12:15:12.594Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}
2022-06-02T12:15:13.238Z        INFO    controller.node Removed emptiness TTL from node {"commit": "10fc37b", "node": "ip-10-132-25-58.eu-central-1.compute.internal"}
2022-06-02T12:15:13.258Z        INFO    controller.volume       Bound persistent volume claim to node ip-10-132-25-58.eu-central-1.compute.internal     {"commit": "10fc37b", "resource": "modulus/redis-data-redis-master-0"}
2022-06-02T12:15:17.483Z        INFO    controller      Batched 4 pod(s) in 1.966634393s        {"commit": "10fc37b"}
2022-06-02T12:15:17.487Z        INFO    controller      0 pod(s) will schedule against new capacity, 4 pod(s) against existing capacity {"commit": "10fc37b"}
2022-06-02T12:15:17.487Z        INFO    controller      Waiting for unschedulable pods  {"commit": "10fc37b"}

Steps to Reproduce the Problem

Resource Specs and Logs

[diranged]

I was just coming in here to open up a similar ticket. We're running the same version of EKS and testing out the same version of Karpenter. We noticed that it didn't seem like Karpenter was able to understand that our EBS CSI Driver has the max volume limit set to 15. I have opened up an issue at kubernetes-sigs/aws-ebs-csi-driver#1258 and also opened up a ticket with our AWS support team - trying to understand if this is a Karpenter bug, or an aws-ebs-csi-driver bug.

Can you check your nodes and see if they already have too many EBS volumes attached?

[tzneal]

@diranged If you look at the output of kubectl describe pod pod-name, what does kube-scheduler show as the reasons why the pod can't schedule?

[diranged]

@diranged If you look at the output of kubectl describe pod pod-name, what does kube-scheduler show as the reasons why the pod can't schedule?

  Warning  FailedScheduling   6s (x2 over 22s)  default-scheduler   0/13 nodes are available: 1 Too many pods, 1 node(s) had taint {kubernetes.io/arch: arm64}, that the pod didn't tolerate, 1 node(s) were unschedulable, 5 node(s) exceed max volume count, 5 node(s) had taint {nd.group-name: kube-system}, that the pod didn't tolerate.

[tzneal]

I think this is a different issue, would you mind creating a new one? The error from this issue doesn't indicate a max volumes per node issue.

[diranged]

@tzneal ok... done in #1888

[sylr]

So it turns out the pod was not scheduled because of unsatisfiable (and bogus) topologySpreadConstraints.

However, karpenter should have been able to detect this and log something about it.

[diranged]

@sylr That reminded me that I wanted to open another feature-enhancement here: #1894

[suket22]

So it turns out the pod was not scheduled because of unsatisfiable (and bogus) topologySpreadConstraints.

Could you give me a little more info on this? Did you have an incorrect value for topologyKey or was your maxSkew a value that could never be satisfied etc?

Fwiw, we'd want Karpenter to conform to what the kube-scheduler would do and not add special logic on top of that, so I'm curious to know what the isue was.

[github-actions]

Labeled for closure due to inactivity in 10 days.