Pod ENI

[bdwyertech]

1. Issue, if available:
#905

2. Description of changes:
This improves instance type selection to factor in Security Groups for Pods by ensuring only instance types capable of Pod ENI are launched if set in a pods resource request.

This leverages the lookup table in vpc-resource-controller to determine Pod ENI support.

3. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 5f43c70

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/61b177e24cceef000741afaa

[ellistarn]

This is a bit challenging. We've been talking about modeling this in a more vendor neutral way like:

Resources() map[resource.Quantity]

I'm willing to refactor this later, and accept this as is, though.

[ellistarn]

This is a pretty cool WIP! I'm assuming that aws.amazon.com/pod-eni is already written onto nodes by the CNI controller and works w/ the kube scheduler?

[bdwyertech]

This is a pretty cool WIP! I'm assuming that aws.amazon.com/pod-eni is already written onto nodes by the CNI controller and works w/ the kube scheduler?

Update:
So digging through the VPC CNI code a bit more, I now understand the logic a bit better. Apparently the label vpc.amazonaws.com/has-trunk-attached=false signals the VPC Resource Controller to attach an ENI.
https://github.com/aws/amazon-vpc-cni-k8s/blob/01654383404fe88f18a89508d9406ad52c6eec1f/pkg/ipamd/ipamd.go#L1194-L1195

There is also a lookup table -- this is ugly, but it is what should be used to determine Pod ENI support:
https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/05e89ff9300a5cc0ebea705834cf27f0a7f3b509/pkg/aws/vpc/limits.go#L27

Also, it looks like there is a well-known limit vpc.amazonaws.com/pod-eni which is added to pods requiring a Pod ENI so the logic here should work fine:

VPC Resource Controller - Pod Webhook

VPC CNI - Handler

[ellistarn]

Awesome! This is shaping up really nicely!

[bdwyertech]

I was kind of surprised by the hard-coded lookup table in the VPC controller and that the new m6i and m6a instances don't support this!

[ellistarn]

Ready for review?

[ellistarn]

One minor comment. Otherwise great job!

[ellistarn]

Nicely done!