Failed to find subsystem mount for required subsystem: pids #16

jt55401 · 2019-04-03T00:29:00Z

Expected Behaviour

kubernetes master node starts and kubectl get pods shows Ready status.

Current Behaviour

kubernetes master node starts, and kubectl get pods shows NotReady status

kubectl describe nodes shows this error in the event log:
Failed to update Node Allocatable Limits ["kubepods"]: failed to set supported cgroup subsystems for cgroup [kubepods]: Failed to find subsystem mount for required subsystem: pids

Possible Solution

not sure - disable whatever requires this cgroup? is it something new in 1.14? or enable that cgroup in rasberian lite somewhere? (I'm not cgroup expert, so I don't know how to even start)

Steps to Reproduce (for bugs)

(follow the guide in this repo, I get these results at the "Check everything worked:" step of the guide)

Context

Can't schedule pods / nodes not ready.

Your Environment

Docker version docker version (e.g. Docker 17.0.05 ):
Docker version 18.09.0, build 4d60db4
What version of Kubernetes are you using? kubectl version:
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/arm"}
Operating System and version (e.g. Linux, Windows, MacOS):
2018-11-13-raspbian-stretch-lite.img
4.14.79-v7+
What ARM or Raspberry Pi board are you using?
Rasberry Pi 3 B+

The text was updated successfully, but these errors were encountered:

shenshouer · 2019-04-27T19:16:45Z

I met the same issue on k8s v1.14.1

hgontijo · 2019-04-29T02:07:58Z

Based on the error message, it looks to me a side effect of Pid limiting, which includes Pod Pids Limit and Node Pids Limit features introduced on v1.14.0, that requires the pids cgroup.

The pids cgroups is not mounted on Raspbian:

$ cat /proc/cgroups
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	8	21	1
cpu	4	86	1
cpuacct	4	86	1
blkio	3	86	1
memory	7	116	1
devices	2	86	1
freezer	6	21	1
net_cls	5	21	1

I tried adding cgroup_enable=pids to /boot/cmdline.txt however it does not mount pids cgroup.
In fact, the kernel configuration file shows that this cgroup is not enabled:

$ uname -ar; zgrep "PIDS" /proc/config.gz
Linux kube-master 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
# CONFIG_CGROUP_PIDS is not set

For now, I see two options:

Use Kubernetes 1.13.x or lower version, or
Recompile Raspbian enabling CONFIG_CGROUP_PIDS

@alexellis, any inputs on that?

bamedro · 2019-04-29T14:15:19Z

Recompiling Raspbian kernel with CONFIG_CGROUP_PIDS enabled fixed this issue for me (I was running k8s v1.14.1)

balboah · 2019-05-08T14:31:47Z

I hit this problem as well, can confirm downgrading to 1.13.5 is working fine.

schachfeld · 2019-05-14T01:11:12Z

I have the same Error. Unfortunatly downgrading does not do the trick for me :(

hgontijo · 2019-05-15T16:48:45Z

PIDs cgroup will be available on the next rpi release: raspberrypi/linux#2968 (comment).

bamedro · 2019-05-15T16:58:16Z

Thanks hgontijo to push for it :)

MichaelSchneeberger · 2019-05-19T08:26:18Z

Is there a good guide/tutorial on how to recompiling Raspbian kernel with CONFIG_CGROUP_PIDS enabled?

MichaelSchneeberger · 2019-05-19T18:04:00Z

Recompiling Raspbian kernel also worked for me. Here is the guide I followed: Kernel building. Add CONFIG_CGROUP_PIDS=y to "arch/arm/configs/bcmrpi_defconfig" (Raspbian source code).

Mike-Dunton · 2019-05-20T14:41:05Z

should a rpi-update fix this yet? Not sure how to tell when the firmware for rpi-4.19.y will be released.

Dionysusio · 2019-05-29T15:15:26Z

I met the same issue on k8s v1.14.1

me too, k8s v1.14.2

staffhorn · 2019-06-11T15:42:35Z

@Mike-Dunton I was able to rpi-update to 4.19.46-v7+ today, and confirm the PIDS fix is in place. kubeadm 1.14.3 installs and inits fine.

davidcollom · 2019-06-23T20:08:44Z

I was able to get my cluster upgraded to 1.15.0 this afternoon by switching the cgroup-driver to systemd.

This does have the issue: Jun 23 20:06:45 rpi-kube-master-01 kubelet[329]: W0623 20:06:45.625225 329 qos_container_manager_linux.go:139] [ContainerManager] Failed to reserve QoS requests: failed to set supported cgroup subsystems for cgroup [kubepods burstable]: Failed to find subsystem mount for required subsystem: pids however kubelet continues to run.

I don't use any burstable pods on my cluster, but until 4.19.46-v7+ becomes stable it may be a work around.

DoGab · 2019-06-28T09:34:37Z

I had the same issue. Did a dist-upgrade from stretch to buster and now kubernetes version v1.15.0 works. Buster has kernel version 4.19.50-v7+ and includes the PIDS Cgroup.
However there is a warning when running kubelet describe node master that says kubelet is posting ready status. WARNING: CPU hardcapping unsupported. I'm not sure if this bothers kubernetes but since it's running fine i guess not.

Note: After upgrading in place i had to disable swap again.

davidcollom · 2019-06-28T13:18:26Z

I've managed to upgrade my raspbian to buster, but it wasn't error free, the following issues where hit:

iptables in nf_tables mode - kube-proxy only works in legacy mode kubernetes/kubernetes#71305 (comment)

swap would be enabled after each boot: sudo systemctl disable dphys-swapfile followed by a reboot, fixed this.

I noticed a substantial amount of errors being reported from docker reporting that cgroupsfs/net_prio being missing (even though it existed and mounted) - upgrading docker-ce to 18.09.0 resolved these - in addition - I had to install docker from stretch repo as busters is known to be broken: docker/for-linux#709

DoGab · 2019-06-28T14:14:15Z

@davidcollom Thank you for those hints. Didn't notice that iptables wasn't working correctly but fixed it now with the command from your link.

alexellis · 2019-06-28T20:54:03Z

Given the issues we are finding with kubeadm on ARMHF, I am thinking about offering an alternate set of instructions to use k3s as per: https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/

What do you think?

davidcollom · 2019-06-28T21:10:23Z

I'm not sure how k3s would solve the issue at hand? Most of these are issues with either kube-proxy (iptables) and low level cgroups being available from the kernel, which were related to raspbian releases.

I've been running my kubeadm cluster for a few months now, and the only issues I find are during upgrades (OS and kubernetes).

The cluster in it's self is stable for day to day operations and requires little/no picking up.

That being said, I have wondered about moving over to k3s as my etcd instance no longer fits along side the control plane and I have a dedicated RPi for etcd.

My cluster consists of 3x amd64, 2x pine64(arm64), 2x RPi "masters" and 7x RPi workers/slaves.

alexellis · 2019-07-12T07:43:48Z

My current recommendation is to use k3s, it uses far fewer resources and works on ARM very well, no timing issues.

https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/

Please try it and let us know if it resolves those issues.

codebykyle · 2019-11-14T11:10:08Z

@Mike-Dunton I was able to rpi-update to 4.19.46-v7+ today, and confirm the PIDS fix is in place. kubeadm 1.14.3 installs and inits fine.

$ sudo rpi-update solved this and made the pids cgroup available, thank you.

alexellis · 2019-12-01T19:44:49Z

Thanks for the comments.

I'm closing / archiving this issue now as it seems to have gone off topic.

My recommendation is that all RPi users try k3s which is GA, compliant and better tested for RPi than kubeadm.

https://blog.alexellis.io/raspberry-pi-homelab-with-k3sup/

alexellis · 2019-12-01T19:45:36Z

PRs are still welcome for the kubeadm guide, just make sure they are tested and have specifics such as versions of Raspbian used and any other steps you ran.

jt55401 changed the title ~~cgroups pids subsystem?~~ Failed to find subsystem mount for required subsystem: pids Apr 3, 2019

hgontijo mentioned this issue May 13, 2019

Enable PIDs cgroup raspberrypi/linux#2968

Closed

HowellBP mentioned this issue Sep 18, 2019

Add CONFIG_CGROUP_PIDS=y? TheRemote/Ubuntu-Server-raspi4-unofficial#21

Closed

petehalverson mentioned this issue Oct 22, 2019

Update swap disable instructions in GUIDE.md #22

Closed

alexellis closed this as completed Dec 1, 2019

Repository owner locked and limited conversation to collaborators Dec 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to find subsystem mount for required subsystem: pids #16

Failed to find subsystem mount for required subsystem: pids #16

jt55401 commented Apr 3, 2019

shenshouer commented Apr 27, 2019

hgontijo commented Apr 29, 2019 •

edited

Loading

bamedro commented Apr 29, 2019

balboah commented May 8, 2019

schachfeld commented May 14, 2019

hgontijo commented May 15, 2019

bamedro commented May 15, 2019

MichaelSchneeberger commented May 19, 2019

MichaelSchneeberger commented May 19, 2019

Mike-Dunton commented May 20, 2019

Dionysusio commented May 29, 2019

staffhorn commented Jun 11, 2019

davidcollom commented Jun 23, 2019 •

edited

Loading

DoGab commented Jun 28, 2019 •

edited

Loading

davidcollom commented Jun 28, 2019

DoGab commented Jun 28, 2019

alexellis commented Jun 28, 2019

davidcollom commented Jun 28, 2019 •

edited

Loading

alexellis commented Jul 12, 2019

codebykyle commented Nov 14, 2019

alexellis commented Dec 1, 2019

alexellis commented Dec 1, 2019

Failed to find subsystem mount for required subsystem: pids #16

Failed to find subsystem mount for required subsystem: pids #16

Comments

jt55401 commented Apr 3, 2019

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

shenshouer commented Apr 27, 2019

hgontijo commented Apr 29, 2019 • edited Loading

bamedro commented Apr 29, 2019

balboah commented May 8, 2019

schachfeld commented May 14, 2019

hgontijo commented May 15, 2019

bamedro commented May 15, 2019

MichaelSchneeberger commented May 19, 2019

MichaelSchneeberger commented May 19, 2019

Mike-Dunton commented May 20, 2019

Dionysusio commented May 29, 2019

staffhorn commented Jun 11, 2019

davidcollom commented Jun 23, 2019 • edited Loading

DoGab commented Jun 28, 2019 • edited Loading

davidcollom commented Jun 28, 2019

DoGab commented Jun 28, 2019

alexellis commented Jun 28, 2019

davidcollom commented Jun 28, 2019 • edited Loading

alexellis commented Jul 12, 2019

codebykyle commented Nov 14, 2019

alexellis commented Dec 1, 2019

alexellis commented Dec 1, 2019

hgontijo commented Apr 29, 2019 •

edited

Loading

davidcollom commented Jun 23, 2019 •

edited

Loading

DoGab commented Jun 28, 2019 •

edited

Loading

davidcollom commented Jun 28, 2019 •

edited

Loading