The AWS VPC CNI has two components. The CNI binary, /opt/cni/bin/aws-cni, and ipamd
running as a Kubernetes daemonset called aws-node, adding a pod on every node that keeps track of all ENIs and IPs attached
to the instance. The CNI binary is invoked by the kubelet when a new pod gets added to, or an existing pod removed from the node.
The CNI binary calls ipamd using gRPC, asking for an IP for the new pod. If there are no available IPs in the cache, it will
return an error. The number of IPs to keep around is controlled by WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET.
The default setting,WARM_ENI_TARGET=1 means that ipamd should keep "a full ENI" of available IPs around. ipamd will
periodically check that it has the right number of available IPs attached, and call EC2 to attach or detach ENIs and IPs when
needed independently of pods being added or deleted on the node.
If the worker nodes are running in small subnets with a limited number of IPs available, using WARM_IP_TARGET together with
MINIMUM_IP_TARGET is an option. MINIMUM_IP_TARGET is the "floor" of how many IPs to keep around on each node. If
you plan to run around 10 pods, the advice is to set MINIMUM_IP_TARGET slightly higher than that, like 12, and have
WARM_IP_TARGET=2.
Some example cases:
| Instance type | WARM_ENI_TARGET |
WARM_IP_TARGET |
MINIMUM_IP_TARGET |
Pods | Attached ENIs | Attached Secondary IPs | Unused IPs | IPs per ENI |
|---|---|---|---|---|---|---|---|---|
| t3.small | 1 | - | - | 0 | 1 | 3 | 3 | 3 |
| t3.small | 1 | - | - | 5 | 3 | 9 | 4 | 3,3,3 |
| t3.small | 1 | - | - | 9 | 3 | 9 | 0 | 3,3,3 |
| t3.small | - | 1 | 1 | 0 | 1 | 1 | 1 | 1 |
| t3.small | - | 1 | 1 | 5 | 2 | 6 | 1 | 3,3 |
| t3.small | - | 1 | 1 | 9 | 3 | 9 | 0 | 3,3,3 |
| t3.small | - | 2 | 5 | 0 | 2 | 5 | 5 | 2,3 |
| t3.small | - | 2 | 5 | 5 | 3 | 7 | 2 | 3,3,1 |
| t3.small | - | 2 | 5 | 9 | 3 | 9 | 0 | 3,3,3 |
| p3dn.24xlarge | 1 | - | - | 0 | 1 | 49 | 49 | 49 |
| p3dn.24xlarge | 1 | - | - | 3 | 2 | 98 | 95 | 49,49 |
| p3dn.24xlarge | 1 | - | - | 95 | 3 | 147 | 52 | 49,49,49 |
| p3dn.24xlarge | - | 5 | 10 | 0 | 1 | 10 | 10 | 10 |
| p3dn.24xlarge | - | 5 | 10 | 7 | 1 | 12 | 5 | 12 |
| p3dn.24xlarge | - | 5 | 10 | 15 | 1 | 20 | 5 | 20 |
| p3dn.24xlarge | - | 5 | 10 | 45 | 2 | 50 | 5 | 49,1 |
To be clear, only set WARM_IP_TARGET for small clusters, or clusters with very low pod churn. It's also advised to set
MINIMUM_IP_TARGET slightly higher than the expected number of pods you plan to run on each node.
The reason to be careful with this setting is that it will increase the number of EC2 API calls that ipamd has to do to attach and detach IPs to the instance. If the number of calls gets too high, they will get throttled and no new ENIs or IPs can be attached to any instance in the cluster.
That is the main reason why we have WARM_ENI_TARGET=1 as the default setting. It's a good balance between having too
many unused IPs attached, and the risk of being throttled by the EC2 API. Creating and attaching a new ENI to a node
can take up to 10 seconds.