Failure in joining worker nodes while creating kind cluster

[leelavg]

What happened:

• Not able to create a cluster with 3 worker nodes:

~/Downloads/k8s $ kind create cluster --image kindest/node:v1.18.2 --config kind-config.yml                                                                                                   
Creating cluster "kind" ...                                                                                                                                                                   
 ✓ Ensuring node image (kindest/node:v1.18.2) 🖼                                                                                                                                               
 ✓ Preparing nodes 📦 📦 📦 📦                                                                                                                                                                
 ✓ Writing configuration 📜                                                                                                                                                                   
 ✓ Starting control-plane 🕹�                                                                                                                                                                   
 ✓ Installing CNI 🔌                                                                                                                                                                          
 ✓ Installing StorageClass 💾                                                                                                                                                                 
 ✗ Joining worker nodes 🚜                                                                                                                                                                    
ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged kind-worker3 kubeadm join --config /kind/kubeadm.conf --ignore-preflight-errors=all --v=6
" failed with error: exit status 1                                                                                                                                                            
Command Output: W0605 03:48:00.642098     459 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.                     
I0605 03:48:00.642208     459 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
[Remaining output truncated]

What you expected to happen:

• kind cluster should be created successfully

How to reproduce it (as minimally and precisely as possible):

• create a tar ball from image kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
• load image from tar ball into docker and tag it appropriately
• create kind cluster $ kind create cluster --image kindest/node:v1.18.2 --config kind-config.yml
• observe the error while joining the worker nodes
• below is the kind-config.yml

apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

Anything else we need to know?:

• able to create cluster with no worker nodes successfully
• able to create cluster with worker nodes before system upgrade
• done a fresh install of golang, docker, kind post upgrade to narrow out any issues due to system upgrade

Environment:

• kind version: (use kind version):
  kind v0.8.1 go1.14.2 linux/amd64

• Kubernetes version: (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Docker version: (use docker info):

 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: /usr/libexec/docker/docker-init
 containerd version: 
 runc version: fbdbaf85ecbc0e077f336c03062710435607dbf1
 init version: 
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 5.6.13-300.fc32.x86_64
 Operating System: Fedora 32 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 14.86GiB
 Name: glv
 ID: SRED:GURW:YTDW:CF6L:TDLK:IGRO:TGGB:TJZC:SAYZ:BXSA:OLCQ:UJ42
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

• OS (e.g. from /etc/os-release):

VERSION="32 (Workstation Edition)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

• GO version:
  go version go1.14.2 linux/amd64

Please let me know if any additional info is required.

[BenTheElder]

Your host may not have enough resources.

Can you create a single cluster with one control plane and one worker?
Can you upload the full logs?

[BenTheElder]

I see you're on fedora 32, Did you look at our open issues at all? #1547

[leelavg]

• before raising the issue I looked at known issues, github issues but searched for joining worker nodes etc
• suspecting this issue is related to recent upgrade, i mentioned related info in questionnaire
• my bad not searching specifically for fedora/OS, anyways thanks for pointing in the right direction and changing firewalld.conf solved the issue.

[timcharper]

For others landing here searching for this issue, it could also be caused by ulimit being too low. See #2744

[MA3CIN]

Thank you @timcharper, the solution you linked worked for me. This issue pops up first if you google "kind failed to join node with kubeadm", so your comment was very helpful!

[strawgate]

A potential sign that this is your issue is that the kind cluster deploys just fine with 1 or 2 worker nodes but fails with 3+ nodes

[nayem9b]

For others landing here searching for this issue, it could also be caused by ulimit being too low. See #2744

Thanks. It really helped. It was a memory issue.