Startup Race Condition

[tsunamishaun]

Version

Karpenter: v0.3.2

Kubernetes: v1.20

Expected Behavior

Scheduling a pod that has a previously attached volume will have the same volume when schedule on a new node provisioned by Karpenter.

Actual Behavior

Pod never gets volume re-attached due to aws-ebs-csi daemon pod not running at time of binding request. It should ultimately retry but this pod (along with aws-vpc-cni pod) are currently path to production for our node startup.

Steps to Reproduce the Problem

Install the aws-ebs-csi driver
Create a pod with a volume attachment
Shift pod somewhere else
Karpenter brings up pod on new node
aws-ebs-csi pod starts up ~1m later

More details in open issue with aws-ebs-csi driver 1030. cc @ellistarn this came out of the meeting sorry for the delay in creating the issue here.

[ellistarn]

We've been exploring this and here's what we've landed on:

• Short term solution: Fix the EBS CSI driver to retry and succeed
• Long term solution: Write a NoAdmit KEP to enforce this ordering with taints in the kubelet (Support Startup Taints #628)

[ellistarn]

Hey @tsunamishaun , I was able to get this working with #1015. Specifically, karpenter needed to apply the SelectedNodeAnnotation, since normally the kube scheduler would do this.

Can you try this in v0.5.4?

[ellistarn]

Closing this as volume support is released.