Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

bagnaram · 2021-05-24T01:04:32Z

What happened:
Preparing Nodes stage fails under rootless Podman.

What you expected to happen:
It should work.

How to reproduce it (as minimally and precisely as possible):

Install podman 3.1.2-2
Install kind v0.11.0 go1.16.4 linux/amd64

Follow steps https://wiki.archlinux.org/title/Podman#Rootless_Podman to enable rootless Podman
Follow steps https://kind.sigs.k8s.io/docs/user/rootless/ to enable rootless Kind

KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✗ Preparing nodes 📦
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume b553212b8e4f27ffdc4913d05e9fa094741bace1b2b64c2b6e31958b4ef27678:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:39369:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:fae9a58f17f18f06aeac9772ca8b5ac680ebbed985e266f711d936e91d113bad" failed with error: exit status 127
Command Output: Error: command rootless-cni-infra [alloc 3dca4d05101ca0d3e9eb0f86c095f8ec2ec4d14d0570c5b44fc630f65c544f5d kind kind-control-plane   ] in container eb38361e52803d8166430e540033d44259e2364ff0c3a77b75f26dfd5b0e9ff7 failed with status 1, stdout="", stderr="failed to list chains: running [/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory\nip6tables v1.8.4 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n\n"

Anything else we need to know?:

Environment:

kind version: kind v0.11.0 go1.16.4 linux/amd64
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"archive", BuildDate:"2021-05-14T14:09:09Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Docker version: (use docker info): podman 3.1.2-2
OS (e.g. from /etc/os-release): 5.12.5-arch1-1

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

The text was updated successfully, but these errors were encountered:

AkihiroSuda · 2021-05-24T06:21:38Z

Please try sudo modprobe ip6tables_nat (https://kind.sigs.k8s.io/docs/user/rootless/)

bagnaram · 2021-05-24T13:03:06Z

The problem is that recent kernel versions do not contain the ip6tables_nat and iptables_nat kernel modules anymore. According to this post, https://bbs.archlinux.org/viewtopic.php?pid=1801151#p1801151 they were remeved at some point from the kernel but I cannot find any secondary source to verify.

Are there any alternatives such as the ones in /usr/lib/modules/5.12.5-arch1-1/kernel/net/ipv6/netfilter/ip6_tables.ko.xz and /usr/lib/modules/5.12.5-arch1-1/kernel/net/ipv4/netfilter/iptable_nat.ko.xz that can be used in lieu? Is there a difference between iptable_natand iptables_nat? I can seemingly run iptables NAT rules on my machine just fine.

bagnaram · 2021-05-24T14:00:49Z

My assumption is that the implementation of iptables between the host machine and the Ubuntu-based 12.0.4 container image is breaking how NAT gets set up.

aojea · 2021-05-24T14:12:10Z

@AkihiroSuda where is this rootless-cni-infra command living?

AkihiroSuda · 2021-05-24T14:44:43Z

https://github.com/containers/podman/tree/v3.1.2/contrib/rootless-cni-infra

aojea · 2021-05-24T14:59:45Z

https://github.com/containers/podman/tree/v3.1.2/contrib/rootless-cni-infra

I can't see those in master, but at first sight it will need this as base container to deal with the iptables/nftables problem and detect the correct backend
https://github.com/kubernetes-sigs/iptables-wrappers

bagnaram · 2021-05-27T15:42:05Z

I added the following file to my local kind node image and I successfully started my kind cluster!

Drop the iptables-wrapper script in images/base/files/usr/local/bin/iptables-wrapper-installer.sh
Created and apply the patch:

From 4a56395a28bd1745781d0dfaa7f8382faf0b55f5 Mon Sep 17 00:00:00 2001
From: bagnaram <[email protected]>
Date: Thu, 27 May 2021 10:34:21 -0500
Subject: [PATCH] podman patches

---
 images/Makefile.common.in | 12 +++++++-----
 images/base/Dockerfile    |  2 ++
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/images/Makefile.common.in b/images/Makefile.common.in
index ee335d1e..1213b4b1 100644
--- a/images/Makefile.common.in
+++ b/images/Makefile.common.in
@@ -14,10 +14,12 @@ export DOCKER_CLI_EXPERIMENTAL=enabled
 
 # build with buildx
 PLATFORMS?=linux/amd64,linux/arm64,linux/ppc64le
+PLATFORMS=linux/amd64
 OUTPUT=
 PROGRESS=auto
-build: ensure-buildx
-	docker buildx build --platform=${PLATFORMS} $(OUTPUT) --progress=$(PROGRESS) -t ${IMAGE} --pull .
+build:
+#	docker buildx build --platform=${PLATFORMS} $(OUTPUT) --progress=$(PROGRESS) -t ${IMAGE} --pull .
+	podman build --platform=${PLATFORMS} $(OUTPUT) -t ${IMAGE} --pull .
 
 # push the cross built image
 push: OUTPUT=--push
@@ -32,7 +34,7 @@ quick: OUTPUT=--load
 quick: build
 
 # enable buildx
-ensure-buildx:
-	./../../hack/build/init-buildx.sh
+#ensure-buildx:
+#	./../../hack/build/init-buildx.sh
 
-.PHONY: push build quick ensure-buildx
\ No newline at end of file
+.PHONY: push build quick ensure-buildx
diff --git a/images/base/Dockerfile b/images/base/Dockerfile
index ec8fcd7c..cf4013ff 100644
--- a/images/base/Dockerfile
+++ b/images/base/Dockerfile
@@ -167,6 +167,8 @@ RUN echo "Installing Packages ..." \
  && echo "Disabling udev" \
     && systemctl disable udev.service
 
+RUN iptables-wrapper-installer.sh
+
 # tell systemd that it is in docker (it will check for the container env)
 # https://systemd.io/CONTAINER_INTERFACE/
 ENV container docker
-- 
2.31.1

I built the node image as shown in this post: Create podmanBuilder for Node Image Builds #2284 (comment)

This is just a test but shows what was necessary to gut it to build. Also buildx appears to be added inline with both docker and podman build commands, so no need to call it explicitly.

What would be the path forward for incorporating the iptables-wrapper? Can it be added as a git submodule or do we copy it as-is to this repository?

aojea · 2021-05-27T15:57:20Z

What would be the path forward for incorporating the iptables-wrapper? Can it be added as a git submodule or do we copy it as-is to this repository?

I reached out to podman developers and they are no longer using that container for podman in master, that image has been deprecated ...

bagnaram · 2021-05-28T00:15:35Z

I actually did not have to touch the rootless-cni-infra container The wrapper script was added to the kind base container image.

BenTheElder · 2021-05-28T00:19:51Z

what's confusing then is kind is doing the iptables-wrapper logic, it just does it in the entrypoint instead of using that project ...

the reason it hasn't worked on podman before is podman was not inserting any iptables rules so there's nothing to select based on. we have other issues where this problem is discussed

BenTheElder · 2021-05-28T00:20:26Z

#1939

bagnaram · 2021-05-28T00:38:11Z

I think my issue mostly deals with a disparity between the version of iptables in the Ubuntu based node image and the iptables kernel modules availible in the bind-mounted /lib/modules directory. The wrapper seemingly updates this.

BenTheElder · 2021-05-28T00:41:54Z

Hmm I don't see the wrapper doing that. https://github.com/kubernetes-sigs/iptables-wrappers/blob/master/iptables-wrapper-installer.sh

We have equivilant simpler logic in the entrypoint given that e.g. we only need to do this for apt not other distro tools.

~~I think it's more likely that we're just not updating all of the binaries correctly in~~

kind/images/base/files/usr/local/bin/entrypoint

Lines 328 to 330 in 227f963

    
           echo "INFO: setting iptables to detected mode: ${mode}" >&2 
        
           update-alternatives --set iptables "/usr/sbin/iptables-${mode}" > /dev/null 
        
           update-alternatives --set ip6tables "/usr/sbin/ip6tables-${mode}" > /dev/null

EDIT: nope, we're effectively doing the same thing as the wrapper 🙃, you don't need to do it for all the binaries, just for the package.

aojea · 2021-05-28T08:01:16Z

I think that The userpace iptables is the one that checks the kernel modules

BenTheElder · 2021-05-28T08:43:02Z

in the wrapper:

if [ "${need_timeout:-0}" = 0 ]; then
    # Write out the simpler version of legacy-vs-nft detection
    cat >> "${sbin}/iptables-wrapper" <<EOF
num_legacy_lines=\$( (iptables-legacy-save || true; ip6tables-legacy-save || true) 2>/dev/null | grep '^-' | wc -l)
num_nft_lines=\$( (iptables-nft-save || true; ip6tables-nft-save || true) 2>/dev/null | grep '^-' | wc -l)
if [ "\${num_legacy_lines}" -ge "\${num_nft_lines}" ]; then
    mode=legacy
else
    mode=nft
fi

in kind:

local mode=nft
  num_legacy_lines=$( (iptables-legacy-save || true; ip6tables-legacy-save || true) 2>/dev/null | grep '^-' | wc -l || true)
  if [ "${num_legacy_lines}" -ge 10 ]; then
    mode=legacy
  else
    num_nft_lines=$( (timeout 5 sh -c "iptables-nft-save; ip6tables-nft-save" || true) 2>/dev/null | grep '^-' | wc -l || true)
    if [ "${num_legacy_lines}" -ge "${num_nft_lines}" ]; then
      mode=legacy
    fi
  fi

BenTheElder · 2021-05-28T08:43:39Z

that logic is not identical. perhaps in podman rootless there are rules from legacy and nft?

BenTheElder · 2021-06-02T19:01:10Z

waiting on discussion of #2289 (comment)
poked sig-networking in Kubernetes slack today.

BenTheElder · 2021-07-29T21:41:39Z

FWIW #2289 landed, I'm still dubious on that being the ultimate root issue.

bagnaram · 2021-07-30T02:30:00Z

It appears to check out fine for me using the latest Kind node images and no longer my custom built ones!

BenTheElder · 2021-08-04T06:12:57Z

Who knew 🙃

This makes me concerned that perhaps podman is resulting in both being used 🤔 but that's not really our problem ...

bagnaram added the kind/bug Categorizes issue or PR as related to a bug. label May 24, 2021

BenTheElder added the area/rootless Issues or PRs related to rootless containers label May 24, 2021

BenTheElder added the area/provider/podman Issues or PRs related to podman label May 25, 2021

BenTheElder mentioned this issue May 28, 2021

match https://github.com/kubernetes-sigs/iptables-wrappers/blob/maste… #2289

Merged

mriedmann mentioned this issue Jun 9, 2021

Error Starting Control Plane w/ Podman on Fedora 33 #2213

Closed

BenTheElder closed this as completed Aug 4, 2021

BenTheElder self-assigned this Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

bagnaram commented May 24, 2021 •

edited

Loading

AkihiroSuda commented May 24, 2021 •

edited

Loading

bagnaram commented May 24, 2021 •

edited

Loading

bagnaram commented May 24, 2021

aojea commented May 24, 2021

AkihiroSuda commented May 24, 2021

aojea commented May 24, 2021

bagnaram commented May 27, 2021 •

edited

Loading

aojea commented May 27, 2021

bagnaram commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented May 28, 2021

bagnaram commented May 28, 2021

BenTheElder commented May 28, 2021 •

edited

Loading

aojea commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented Jun 2, 2021

BenTheElder commented Jul 29, 2021

bagnaram commented Jul 30, 2021 •

edited

Loading

BenTheElder commented Aug 4, 2021

Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

Comments

bagnaram commented May 24, 2021 • edited Loading

AkihiroSuda commented May 24, 2021 • edited Loading

bagnaram commented May 24, 2021 • edited Loading

bagnaram commented May 24, 2021

aojea commented May 24, 2021

AkihiroSuda commented May 24, 2021

aojea commented May 24, 2021

bagnaram commented May 27, 2021 • edited Loading

aojea commented May 27, 2021

bagnaram commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented May 28, 2021

bagnaram commented May 28, 2021

BenTheElder commented May 28, 2021 • edited Loading

aojea commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented May 28, 2021

BenTheElder commented Jun 2, 2021

BenTheElder commented Jul 29, 2021

bagnaram commented Jul 30, 2021 • edited Loading

BenTheElder commented Aug 4, 2021

bagnaram commented May 24, 2021 •

edited

Loading

AkihiroSuda commented May 24, 2021 •

edited

Loading

bagnaram commented May 24, 2021 •

edited

Loading

bagnaram commented May 27, 2021 •

edited

Loading

BenTheElder commented May 28, 2021 •

edited

Loading

bagnaram commented Jul 30, 2021 •

edited

Loading