Support IPv6 kubernetes cluster When using generated Launch Template

[literalice]

Tell us about your request

When using generated LaunchTemplate and IPv6 cluster it cannot provision nodes normally.

• Node internal IP is associated to IPv4 address.
• Service IPv6 CIDR is not specified to kubelet.

In case of clusters on AWS, optimized AMI has the options for setting node up for IPv6 cluster.

It's very useful to use these options in the Launch Template generated by Karpenter.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
The users can use custom Launch Template and userdata that call bootstrap script. However I think preparing it only for launching nodes in IPv6 cluster seems to be annoying.

Are you currently working around this issue?
As mentioned above custom Launch Template and userdata can be a work-around.

Additional context
I prepared a PR for the issue: #1232 .
It seems reasonable to me to implement this as options of Karpenter controller. Could you please tell me your opnions and advices

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[youwalther65]

The following request would help here as well by querying the EKS DescribeCluster API for provider AWS. As a current workaround what's about using a custom LaunchTemplate which provides "--ip-family ipv6 --service-ipv6-cidr cluster-ipv6-cidr" to bootstrap.sh here

[ellistarn]

We intentionally do not rely on EKS to ensure that Karpenter works well on selfhosted AWS (e.g. CAPA, etc). It's potentially a candidate for autodiscovery, but I'm wary about the number of code paths this creates.

[ellistarn]

How does ipv6 work for bottlerocket?

[mKeRix]

FWIW I have an IPv6 EKS cluster with Karpenter managed worker nodes functioning correctly. I’m using the Bottlerocket amiFamily. I did have to manually set the cluster DNS in the provisioner kubeletConfiguration to the IPv6 DNS service address, but the rest worked out of the box.

[ellistarn]

I'd love to see this feature driven via the existing kubeletconfiguration.ClusterDNS field. We can infer ipv4 vs ipv6 from this

[youwalther65]

We intentionally do not rely on EKS to ensure that Karpenter works well on selfhosted AWS (e.g. CAPA, etc). It's potentially a candidate for autodiscovery, but I'm wary about the number of code paths this creates.

Good point, agree. Maybe one of the EKS tags can help differentiate between AWS managed (EKS) and unmanaged K8s?!

[youwalther65]

But the nodes created with such a config:
$ cat provisioner.yaml

---
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  limits:
    resources:
      cpu: 1000
  provider:
    securityGroupSelector:
      kubernetes.io/cluster/my-ipv6-cluster: '*'
    subnetSelector:
      kubernetes.io/cluster/my-ipv6-cluster: '*'
  requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - on-demand
  kubeletConfiguration:
    clusterDNS: ["fd13:b96c:ce20::a"]
  ttlSecondsAfterEmpty: 30

do not register in EKS cluster with their IPv6 address:
$ k get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-124-218.eu-west-1.compute.internal Ready 20m v1.21.5-eks-9017834 2a05:d018:1916:aa03:47d5:1b28:2101:a36d Amazon Linux 2 5.4.176-91.338.amzn2.x86_64 docker://20.10.7
ip-192-168-163-72.eu-west-1.compute.internal Ready 20m v1.21.5-eks-9017834 2a05:d018:1916:aa05:5f45:d01a:e0ec:4c63 Amazon Linux 2 5.4.176-91.338.amzn2.x86_64 docker://20.10.7
ip-192-168-70-31.eu-west-1.compute.internal Ready 3m26s v1.21.5-eks-9017834 192.168.70.31 3.249.223.152 Amazon Linux 2 5.4.176-91.338.amzn2.x86_64 containerd://1.4.6

So kubeletConfiguration need some more options to provide ipFamily. @ellistarn Is this alreayd available/implemented

In addition I wonder that the Karpenter node is provisioned containerd alreayd. Was this changed in AMI or does eksctl 0.86.0-rc.0 does some "magic" here? Karpenter section in eksctl first works with this - see issue

[youwalther65]

A working IPv6 nodes registers with its IPv6 provided to kubelet via KUBELET_ARGS "--node-ip"

$ pwd
/etc/systemd/system/kubelet.service.d

$ cat 10-kubelet-args.conf
[Service]
Environment='KUBELET_ARGS=--node-ip=2a05:d018:1916:aa03:47d5:1b28:2101:a36d --pod-infra-container-image=602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/pause:3.1-eksbuild.1 --v=2'

This is set in /etc/eks/bootstrap.sh" via:
412 if [[ "${IP_FAMILY}" == "ipv4" ]]; then
413 INTERNAL_IP=$(get_meta_data 'latest/meta-data/local-ipv4')
414 else
415 INTERNAL_IP_URI=latest/meta-data/network/interfaces/macs/$MAC/ipv6s
416 INTERNAL_IP=$(get_meta_data $INTERNAL_IP_URI)
417 fi

or direct link So we "just" need to pass "--ip-family ipv6" to "bootstrap.sh" to get it working as expected.

/bootstrap.sh --help
usage: ./bootstrap.sh [options]
Bootstraps an instance into an EKS cluster

...
--ip-family Specify ip family of the cluster
--service-ipv6-cidr ipv6 cidr range of the cluster

service-ipv6-cidr is "only" used to automaticall set the DNS serverIPv6 by adding a trailing "a" as address which provisioner already sets via clusterDNS

And as @ellistarn already proposed one could infer the IP family type from the DNS server address type. But it would be great to see Karpenter figures out the kube-dsn service IP automatically so we can avoid "tweaking" the provisioner configuration for that.

[literalice]

Hi, currently I proposed the PR which infers the cluster IP family from cluster DNS parameter.
It has no change in the user facing Provisioner API so the Karpenter users simply can get benefit that they can register a provisioning node as IPv6 one.

But it would be great to see Karpenter figures out the kube-dsn service IP automatically so we can avoid "tweaking" the provisioner configuration for that.

Thank you for the comment, so we may to be able to use "kubernetes.default.svc" to infer IP family.

[ellistarn]

Thank you for the comment, so we may to be able to use "kubernetes.default.svc" to infer IP family.

This would be a huge value add. Perhaps we can discover it if not specified in kubeletconfiguration. And then we can deprecate it in v1alpha6.

[billrayburn]

Closing, going out in next release.