Document Pod Density and Default Instance Profile

[prateekgogia]

Is an existing page relevant?

What karpenter features are relevant?

#1032
#914

How should the docs be improved?
For the issues linked above, Karpenter is missing docs update

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[mikesir87]

I'd be happy to take this on, since I authored the two mentioned PRs.

@ellistarn We don't have a good spot in the docs for CLI flags. Add a new page or add it to an existing one? In this case, both are AWS-specific too. Maybe a single CLI flag page with different sections for the various providers?

[ellistarn]

@geoffcline @chrisnegus can you chime in on structure?

[geoffcline]

Suggestion: Add "reference" section, move provisioner API under new section; add karpenter args page to this section; identify AWS specific content using a "☁️ AWS" header. This is assuming the page would be <50% AWS specific content.

[alekc]

Since we are at it, I would suggest adding a separate value in helm chart for this env var, something like

aws:
  # -- The default instance profile to use when launching nodes on AWS
  defaultInstanceProfile: ""
  eniLimitedPodDensity: true

which if set to false will add envvar to the controller deployment

[geoffcline]

@ellistarn can I get a 2-3 sentence summary of #1032 and this issue?

[ellistarn]

Nodes are configured support a maximum number of pods, specific to the cloud provider. Previously for AWS, this number was limited by the number of attachable ENIs on the specific EC2 instance type. With AWS VPC CNI v1.9+, much higher pod densities are supported. By default, --aws-eni-limited-pod-density=false, the AWS cloud provider will limit the number of pods to the default Kubernetes upstream value of 110.

[alekc]

@ellistarn

By default, --aws-eni-limited-pod-density=false, the AWS cloud provider will limit the number of pods to the default Kubernetes upstream value of 110.

I might be misreading what you wrote, but the --aws-eni-limited-pod-density is true by default (https://github.com/aws/karpenter/blob/5a5d01572480f54f7c0aad1f0238ec078a958319/pkg/utils/options/options.go#L43), thus limiting pod capacity to 11 pods in case of t3.small

How about something along the lines "To support higher density, set --aws-eni-limited-pod-density=true (or env var AWS_ENI_LIMITED_POD_DENSITY=true) and change your cni configuration as per aws documentation https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

[github-actions]

This issue is stale because it has been open 25 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[geoffcline]

closed with #1237