Nodes NotReady when selecting containerd runtime

[ivallhon]

What happened:

When bootstrapping an EKS node with --container-runtime containerd nodes are in NotReady status

What you expected to happen:

Nodes are functional to the cluster

How to reproduce it (as minimally and precisely as possible):

EKS_VERSION=1.21

AMI_ID=$(aws ssm get-parameter \
    --name /aws/service/eks/optimized-ami/${EKS_VERSION}/amazon-linux-2/recommended/image_id \
    --query "Parameter.Value" --output text)

AWS_REGION=${AWS_DEFAULT_REGION:-us-west-2}

CLUSTER_NAME=containerd-eks

cat > eksctl-containerd.yaml <<EOF
--- 
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_REGION}
  version: "${EKS_VERSION}"
managedNodeGroups:
  - name: containerd
    ami: ${AMI_ID}
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd
EOF

eksctl create cluster -f eksctl-containerd.yaml

Instructions from: https://aws.amazon.com/blogs/containers/amazon-eks-1-21-released/

Anything else we need to know?:

The symbolic link of /run/containerd/containerd.sock to /run/dockershim.sock that bootstrap.sh creates is not created.

Running ln -sf /run/containerd/containerd.sock /run/dockershim.sock and starting kubelet gets the node in Ready state

--logs--

Jan 12 17:59:17 ip-192-168-87-98 systemd: Started Execute cloud user/final scripts.
Jan 12 17:59:18 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 3840ms.
Jan 12 17:59:20 ip-192-168-87-98 chronyd[2901]: Selected source 169.254.169.123
Jan 12 17:59:22 ip-192-168-87-98 systemd: Started Dynamically Generate Message Of The Day.
Jan 12 17:59:22 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 7850ms.
Jan 12 17:59:27 ip-192-168-87-98 pull-sandbox-image.sh: ctr: failed to dial "/run/dockershim.sock": context deadline exceeded
Jan 12 17:59:27 ip-192-168-87-98 systemd: Removed slice User Slice of root.
Jan 12 17:59:30 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 15290ms.
Jan 12 17:59:42 ip-192-168-87-98 pull-sandbox-image.sh: Attempt 1 of 5
Jan 12 17:59:42 ip-192-168-87-98 systemd: Created slice User Slice of root.
Jan 12 17:59:42 ip-192-168-87-98 systemd: Started Session c2 of user root.
Jan 12 17:59:45 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 30440ms.
Jan 12 17:59:52 ip-192-168-87-98 pull-sandbox-image.sh: ctr: failed to dial "/run/dockershim.sock": context deadline exceeded
Jan 12 17:59:52 ip-192-168-87-98 systemd: Removed slice User Slice of root.
Jan 12 18:00:16 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 62760ms.
Jan 12 18:00:21 ip-192-168-87-98 pull-sandbox-image.sh: Attempt 2 of 5
Jan 12 18:00:21 ip-192-168-87-98 systemd: Created slice User Slice of root.
Jan 12 18:00:21 ip-192-168-87-98 systemd: Started Session c3 of user root.
Jan 12 18:00:31 ip-192-168-87-98 pull-sandbox-image.sh: ctr: failed to dial "/run/dockershim.sock": context deadline exceeded
Jan 12 18:00:31 ip-192-168-87-98 systemd: Removed slice User Slice of root.
Jan 12 18:01:01 ip-192-168-87-98 systemd: Created slice User Slice of root.
Jan 12 18:01:01 ip-192-168-87-98 systemd: Started Session 1 of user root.
Jan 12 18:01:12 ip-192-168-87-98 pull-sandbox-image.sh: Attempt 3 of 5
Jan 12 18:01:12 ip-192-168-87-98 systemd: Started Session c4 of user root.
Jan 12 18:01:19 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 108790ms.
Jan 12 18:01:22 ip-192-168-87-98 pull-sandbox-image.sh: ctr: failed to dial "/run/dockershim.sock": context deadline exceeded
Jan 12 18:02:48 ip-192-168-87-98 pull-sandbox-image.sh: Attempt 4 of 5
Jan 12 18:02:48 ip-192-168-87-98 systemd: Started Session c5 of user root.
Jan 12 18:02:58 ip-192-168-87-98 pull-sandbox-image.sh: ctr: failed to dial "/run/dockershim.sock": context deadline exceeded
Jan 12 18:03:08 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 126420ms.
Jan 12 18:05:14 ip-192-168-87-98 dhclient[3248]: XMT: Solicit on eth0, interval 108070ms.

Environment:

• AWS Region: us-west-2
• Instance Type(s):
• EKS Platform version : eks.4
• Kubernetes version: 1.21
• AMI Version: amazon-eks-node-1.21-v20211206
• Kernel (e.g. uname -a): Linux ip-192-168-87-98.us-west-2.compute.internal 5.4.156-83.273.amzn2.x86_64
• Release information (run cat /etc/eks/release on a node):
  BASE_AMI_ID="ami-051e829ae81b0b8db"
  BUILD_TIME="Mon Dec 6 20:08:18 UTC 2021"
  BUILD_KERNEL="5.4.156-83.273.amzn2.x86_64"
  ARCH="x86_64"

[ivallhon]

This is resolved on the latest EKS-Optimized AMI: https://github.com/awslabs/amazon-eks-ami/releases/tag/v20220112