containerd+driver none returning error #13821

staticdev · 2022-03-21T11:55:48Z

What Happened?

Also worth noticing iptables is installed and on PATH.

Logs:

 % minikube start --driver=none --container-runtime=containerd --alsologtostderr
I0321 12:53:27.255333    3949 out.go:297] Setting OutFile to fd 1 ...
I0321 12:53:27.255390    3949 out.go:349] isatty.IsTerminal(1) = true
I0321 12:53:27.255394    3949 out.go:310] Setting ErrFile to fd 2...
I0321 12:53:27.255400    3949 out.go:349] isatty.IsTerminal(2) = true
I0321 12:53:27.255462    3949 root.go:315] Updating PATH: /home/user/.minikube/bin
I0321 12:53:27.255683    3949 out.go:304] Setting JSON to false
I0321 12:53:27.318459    3949 start.go:112] hostinfo: {"hostname":"host","uptime":166,"bootTime":1647863441,"procs":379,"os":"linux","platform":"debian","platformFamily":"debian","platformVersion":"11.2","kernelVersion":"5.10.0-12-amd64","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"19beb2c0-7a7f-4725-8977-b58ef5a8f577"}
I0321 12:53:27.318524    3949 start.go:122] virtualization: kvm host
I0321 12:53:27.319997    3949 out.go:176] 😄  minikube v1.25.2 on Debian 11.2
😄  minikube v1.25.2 on Debian 11.2
I0321 12:53:27.320714    3949 notify.go:193] Checking for updates...
I0321 12:53:27.321234    3949 driver.go:344] Setting default libvirt URI to qemu:///system
I0321 12:53:27.322283    3949 out.go:176] ✨  Using the none driver based on user configuration
✨  Using the none driver based on user configuration
I0321 12:53:27.322291    3949 start.go:281] selected driver: none
I0321 12:53:27.322295    3949 start.go:798] validating driver "none" against <nil>
I0321 12:53:27.322303    3949 start.go:809] status for none: {Installed:false Healthy:false Running:true NeedsImprovement:false Error:exec: "iptables": executable file not found in $PATH Reason: Fix:iptables must be installed Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/none/ Version:}
I0321 12:53:27.324163    3949 out.go:176] 

W0321 12:53:27.324208    3949 out.go:241] 🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "iptables": executable file not found in $PATH
🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "iptables": executable file not found in $PATH
W0321 12:53:27.324245    3949 out.go:241] 💡  Suggestion: iptables must be installed
💡  Suggestion: iptables must be installed
W0321 12:53:27.324285    3949 out.go:241] 📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
I0321 12:53:27.325088    3949 out.go:176]

Attach the log file

 % minikube logs --file=log.txt
🤷  Profile "minikube" not found. Run "minikube profile list" to view all profiles.
👉  To start a cluster, run: "minikube start"

Operating System

Debian

Driver

No response

The text was updated successfully, but these errors were encountered:

afbjorklund · 2022-03-21T11:59:12Z

What does which iptables say ? It is usually found in /usr/sbin/iptables.

vagrant@ubuntu2004:~$ which iptables
/usr/sbin/iptables
vagrant@ubuntu2004:~$ iptables --version
iptables v1.8.4 (legacy)
vagrant@ubuntu2004:~$ readlink -f /usr/sbin/iptables
/usr/sbin/xtables-legacy-multi
vagrant@ubuntu2004:~$ dpkg -S /usr/sbin/xtables-legacy-multi
iptables: /usr/sbin/xtables-legacy-multi
vagrant@ubuntu2004:~$ apt list iptables
Listing... Done
iptables/focal,now 1.8.4-3ubuntu2 amd64 [installed]
iptables/focal 1.8.4-3ubuntu2 i386

BTW. As part of the Kubernets requirements, you also need to install "conntrack".

afbjorklund · 2022-03-21T12:02:41Z

Here is the configuration that I was testing with:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2004"

  config.vm.provider "libvirt" do |lv|
     lv.cpus = 2
     lv.memory = 2048
  end

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y containerd conntrack
    ln -s /bin/false /usr/local/bin/docker
    # cri-tools
    wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-linux-amd64.tar.gz
    tar -C /usr/local/bin -xzf crictl-v1.23.0-linux-amd64.tar.gz
    # cni-plugins
    wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
    mkdir -p /opt/cni /opt/cni/bin
    chmod 755 /opt/cni /opt/cni/bin
    tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.1.1.tgz
    # minikube
    wget https://github.com/kubernetes/minikube/releases/download/v1.25.2/minikube_1.25.2-0_amd64.deb
    dpkg -i minikube_1.25.2-0_amd64.deb
  SHELL
end

minikube start --driver=none --container-runtime=containerd --cni=bridge

😄  minikube v1.25.2 on Ubuntu 20.04 (kvm/amd64)
✨  Using the none driver based on user configuration
❗  Using the 'containerd' runtime with the 'none' driver is an untested configuration!

🧯  The requested memory allocation of 1983MiB does not leave room for system overhead (total system memory: 1983MiB). You may face stability issues.
💡  Suggestion: Start minikube with less memory allocated: 'minikube start --memory=1983mb'

👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=2, Memory=1983MB, Disk=126099MB) ...
ℹ️  OS release is Ubuntu 20.04.4 LTS
📦  Preparing Kubernetes v1.23.3 on containerd 1.5.5-0ubuntu3 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/vagrant
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/vagrant/.kube /home/vagrant/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

staticdev · 2022-03-21T12:12:25Z

About iptables:

$ iptables --version
iptables v1.8.7 (nf_tables)

$ readlink -f /usr/sbin/iptables
/usr/sbin/xtables-nft-multi

$ dpkg -S /usr/sbin/xtables-legacy-multi
iptables: /usr/sbin/xtables-legacy-multi

$ apt list iptables  
Listing... Done
iptables/stable,now 1.8.7-1 amd64 [installed]

$ ls -l /usr/sbin|grep iptables
lrwxrwxrwx 1 root root        14 Jan 17  2021 ip6tables-apply -> iptables-apply
lrwxrwxrwx 1 root root        26 Aug 19  2021 iptables -> /etc/alternatives/iptables
-rwxr-xr-x 1 root root      7057 Jan 15  2021 iptables-apply
lrwxrwxrwx 1 root root        20 Jan 17  2021 iptables-legacy -> xtables-legacy-multi
lrwxrwxrwx 1 root root        20 Jan 17  2021 iptables-legacy-restore -> xtables-legacy-multi
lrwxrwxrwx 1 root root        20 Jan 17  2021 iptables-legacy-save -> xtables-legacy-multi
lrwxrwxrwx 1 root root        17 Jan 17  2021 iptables-nft -> xtables-nft-multi
lrwxrwxrwx 1 root root        17 Jan 17  2021 iptables-nft-restore -> xtables-nft-multi
lrwxrwxrwx 1 root root        17 Jan 17  2021 iptables-nft-save -> xtables-nft-multi
lrwxrwxrwx 1 root root        34 Aug 19  2021 iptables-restore -> /etc/alternatives/iptables-restore
lrwxrwxrwx 1 root root        17 Jan 17  2021 iptables-restore-translate -> xtables-nft-multi
lrwxrwxrwx 1 root root        31 Aug 19  2021 iptables-save -> /etc/alternatives/iptables-save
lrwxrwxrwx 1 root root        17 Jan 17  2021 iptables-translate -> xtables-nft-multi

@afbjorklund can you try with config.vm.box = "generic/debian11"

generic/debian11

afbjorklund · 2022-03-21T12:18:34Z

It seems like the vagrant box for debian11 does not include "iptables".

apt install -y iptables

And regular users on Debian, they do not have /usr/sbin in PATH.

vagrant@debian11:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

I think minikube runs it with "sudo" anyway, so should look in root PATH.

vagrant@debian11:~$ iptables
-bash: iptables: command not found
vagrant@debian11:~$ sudo iptables
iptables v1.8.7 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.

staticdev · 2022-03-21T12:19:58Z

Update: besides being installed /usr/sbin doesn't seem to be on default path of debian. I exported the path manually.. and the logs change to complain about docker:

minikube start --driver=none --container-runtime=containerd --alsologtostderr
I0321 13:17:04.297447    5009 out.go:297] Setting OutFile to fd 1 ...
I0321 13:17:04.297522    5009 out.go:349] isatty.IsTerminal(1) = true
I0321 13:17:04.297532    5009 out.go:310] Setting ErrFile to fd 2...
I0321 13:17:04.297538    5009 out.go:349] isatty.IsTerminal(2) = true
I0321 13:17:04.297626    5009 root.go:315] Updating PATH: /home/static/.minikube/bin
I0321 13:17:04.297840    5009 out.go:304] Setting JSON to false
I0321 13:17:04.338891    5009 start.go:112] hostinfo: {"hostname":"host","uptime":1583,"bootTime":1647863441,"procs":354,"os":"linux","platform":"debian","platformFamily":"debian","platformVersion":"11.2","kernelVersion":"5.10.0-12-amd64","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"19beb2c0-7a7f-4725-8977-b58ef5a8f577"}
I0321 13:17:04.338976    5009 start.go:122] virtualization: kvm host
I0321 13:17:04.340447    5009 out.go:176] 😄  minikube v1.25.2 on Debian 11.2
😄  minikube v1.25.2 on Debian 11.2
I0321 13:17:04.340617    5009 notify.go:193] Checking for updates...
I0321 13:17:04.340648    5009 driver.go:344] Setting default libvirt URI to qemu:///system
I0321 13:17:04.342194    5009 out.go:176] ✨  Using the none driver based on user configuration
✨  Using the none driver based on user configuration
I0321 13:17:04.342209    5009 start.go:281] selected driver: none
I0321 13:17:04.342217    5009 start.go:798] validating driver "none" against <nil>
I0321 13:17:04.342231    5009 start.go:809] status for none: {Installed:false Healthy:false Running:true NeedsImprovement:false Error:exec: "docker": executable file not found in $PATH Reason: Fix:Install docker Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/none/ Version:}
I0321 13:17:04.347834    5009 out.go:176] 

W0321 13:17:04.347906    5009 out.go:241] 🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "docker": executable file not found in $PATH
🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "docker": executable file not found in $PATH
W0321 13:17:04.347993    5009 out.go:241] 💡  Suggestion: Install docker
💡  Suggestion: Install docker
W0321 13:17:04.348040    5009 out.go:241] 📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
I0321 13:17:04.349558    5009 out.go:176]

Now I see what you mentioned on Slack...

afbjorklund · 2022-03-21T12:23:35Z

The "docker" issue was known, unfortunately.

ln -s /bin/false /usr/local/bin/docker

none with containerd: make it possible to run without the docker CLI #10908
none driver can't be used with non-Docker runtimes (looks for "docker" executable) #5549

staticdev · 2022-03-21T13:12:26Z

After adding this fake docker command and installing conntrack, I now have an error of missing crictl:

l0321 14:10:40.869553    7144 exec_runner.go:51] Run: sudo crictl version
I0321 14:10:40.873750    7144 retry.go:31] will retry after 11.04660288s: Temporary Error: sudo crictl version: exit status 1
stdout:

stderr:
sudo: crictl: command not found
I0321 14:10:51.921112    7144 exec_runner.go:51] Run: sudo crictl version
I0321 14:10:51.926696    7144 retry.go:31] will retry after 21.607636321s: Temporary Error: sudo crictl version: exit status 1
stdout:

stderr:
sudo: crictl: command not found
I0321 14:11:13.536806    7144 exec_runner.go:51] Run: sudo crictl version
I0321 14:11:13.541835    7144 retry.go:31] will retry after 26.202601198s: Temporary Error: sudo crictl version: exit status 1
stdout:

stderr:
sudo: crictl: command not found
I0321 14:11:39.748017    7144 exec_runner.go:51] Run: sudo crictl version
I0321 14:11:39.759375    7144 out.go:176] 

W0321 14:11:39.759511    7144 out.go:241] ❌  Exiting due to RUNTIME_ENABLE: Temporary Error: sudo crictl version: exit status 1
stdout:

stderr:
sudo: crictl: command not found

❌  Exiting due to RUNTIME_ENABLE: Temporary Error: sudo crictl version: exit status 1
stdout:

stderr:
sudo: crictl: command not found

W0321 14:11:39.759532    7144 out.go:241] 

W0321 14:11:39.760648    7144 out.go:241] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I0321 14:11:39.761511    7144 out.go:176]

afbjorklund · 2022-03-21T13:16:48Z

When you use the none driver, then you are reponsible for installing the Kubernetes requirements such as CRI and CNI.

/usr/bin/crictl

/opt/cni/bin/*

There are some shell script examples above, the k8s.io documentation upstream on these installations is not very great.

Minikube will set up the configuration, but not install the programs...

For the supported OS, they both come pre-installed with the system.

afbjorklund · 2022-03-22T06:34:29Z

The bug report itself is invalid, since iptables was not on the PATH. On the other hand, it doesn't have to be either...

Minikube should check that it is available to root, either by hardcoding "/usr/sbin/iptables" or by running "sudo iptables".

staticdev · 2022-03-22T20:50:56Z

@afbjorklund you are right. Problem is the same as the ones you mentioned using docker. closing here. Thanks a lot for the help!

nagstaku · 2023-04-13T19:18:58Z

also ran into the problem of needing to stub docker to /bin/false :(

afbjorklund added co/none-driver co/runtime/containerd labels Mar 21, 2022

staticdev changed the title ~~containerd+driver none returning error on iptables~~ containerd+driver none returning error Mar 21, 2022

afbjorklund added kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Mar 21, 2022

afbjorklund added priority/backlog Higher priority than priority/awaiting-more-evidence. os/linux and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Mar 22, 2022

staticdev closed this as completed Mar 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containerd+driver none returning error #13821

containerd+driver none returning error #13821

staticdev commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

staticdev commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

staticdev commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 •

edited

Loading

staticdev commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 22, 2022 •

edited

Loading

staticdev commented Mar 22, 2022

nagstaku commented Apr 13, 2023

containerd+driver none returning error #13821

containerd+driver none returning error #13821

Comments

staticdev commented Mar 21, 2022 • edited Loading

What Happened?

Attach the log file

Operating System

Driver

afbjorklund commented Mar 21, 2022 • edited Loading

afbjorklund commented Mar 21, 2022 • edited Loading

staticdev commented Mar 21, 2022 • edited Loading

afbjorklund commented Mar 21, 2022 • edited Loading

staticdev commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 • edited Loading

staticdev commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 • edited Loading

afbjorklund commented Mar 22, 2022 • edited Loading

staticdev commented Mar 22, 2022

nagstaku commented Apr 13, 2023

staticdev commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

staticdev commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 22, 2022 •

edited

Loading