Karpenter fails to create node object if an instance does not have PrivateDnsName still an issue

[bwmetcalf]

Version

Karpenter: v0.5.4

Kubernetes: v1.21.5

Actual Behavior

We have seen this issue twice in the past week running 0.5.4. According to #1067, the issue is fixed in 0.5.4.

#1: got instance i-0eaa445d1c9aa01bb but PrivateDnsName was not set
#2: got instance i-0eaa445d1c9aa01bb but PrivateDnsName was not set
#3: got instance i-0eaa445d1c9aa01bb but PrivateDnsName was not set     {"commit": "7e79a67", "provisioner": "gitlab-runner-provisioner-karpenter-provisioner"}

Resource Specs and Logs

Same issue as #1067

[ellistarn]

To be clear from that issue, this is an unfortunate race condition that EC2 sometimes takes a long time to propagate the private dns name. If you're using karpenter owned launch templates, it should still register and work since karpenter injects all the expected labels into the userdata. If you're using a custom launch template, make sure you include karpenter.sh/provisioner-name as a label passed to the kubelet.

I think we could explore increasing the retries/timeout on EC2, as well.

[felix-zhe-huang]

Is there a known way to reproduce this bug constantly?

[felix-zhe-huang]

PR #1286 doubles the get instance retry to 6 times, one second apart.
This will double the time that a private DNS name can be assigned before Karpenter raises an error.