Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" #10371

costiser · 2021-02-05T10:38:56Z

Steps to reproduce the issue:

clean install of docker and clean install minikube
docker group created and user added to it (as instructed here: https://docs.docker.com/engine/install/linux-postinstall/)
3.minikube start --driver=docker fails

Full output of failed command:

Full output of minikube start command used, if not already included:

Command: minikube start --driver=docker:

😄  minikube v1.17.0 on Debian rodete
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🎉  minikube 1.17.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.17.1
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0.scope/memory.swap.max: permission denied: unknown.

🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max: permission denied: unknown.


❌  Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max: permission denied: unknown.


😿  If the above advice does not help, please let us know:
👉  https://github.com/kubernetes/minikube/issues/new/choose

Optional: Full output of minikube logs command:

🤷 The control plane node "" does not exist. 👉 To start a cluster, run: "minikube start"

The text was updated successfully, but these errors were encountered:

afbjorklund · 2021-02-05T12:35:15Z

I think you need to configure those in the Kernel settings, when running on Debian ?

https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

Currently we only check for "memsw.limit_in_bytes", but not for "memory.swap.max"

There should be a similar check to the one for cgroups v1, also for the new cgroups v2.

costiser · 2021-02-05T20:27:37Z

Has this changed recently? I'm trying to figure out what changed recently, because minikube was working fine on the same machine before upgrading minikube & docker...
do you plan to add checks also for memory.swap.max (similar to what you mentioned "memsw.limit_in_bytes")?

awan1 · 2021-02-05T22:22:13Z

I ran into this problem today too, also after a restart, before which everything was working fine. Making the GRUB changes listed in the docs linked by @afbjorklund made it work. Thank you!

Edit: now that I'm thinking about it, I reinstalled Minikube because I thought maybe I needed to update to a new version, so it makes sense that I needed to follow the post-install instructions.

Edit 2: maybe part of the trouble here is that the error message did not contain the message indicated in the docs, "Your kernel does not support cgroup swap limit capabilities". Maybe the error message could be updated to be closer to that, or the docs could include a sample of the error message we encountered.

afbjorklund · 2021-02-05T22:50:16Z

Has this changed recently? I'm trying to figure out what changed recently, because minikube was working fine on the same machine before upgrading minikube & docker...

Older versions of Docker (before 20.10) did not support cgroups v2.

The kernel setting is the same, though. It's something specific to Debian, others have it on by default...

do you plan to add checks also for memory.swap.max (similar to what you mentioned "memsw.limit_in_bytes")?

Yes.

In this section: https://github.com/kubernetes/minikube/blob/v1.17.1/pkg/drivers/kic/oci/oci.go#L157_L161

costiser · 2021-02-06T21:25:11Z

I confirm it's working now with the changes suggested on https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

medyagh · 2021-02-12T21:59:18Z

I confirm this is a problem that affects users with cgroup2

and one thing we could do on minikube side is NOT applying memory limits for cgroup v2 until it is fixed and also point them to the docs on Docker website to fix their cgroup

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
 
Server:
 Containers: 16
  Running: 0
  Paused: 0
  Stopped: 16
 Images: 21
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.7.17-1rodete5-amd64
 Operating System: Debian GNU/Linux rodete
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 125.5GiB
 Name: drewpca.sbo.corp.google.com
 ID: F4RJ:PGBV:QUDU:5P77:Y3AV:YEKE:RWX3:T6QM:MPCP:WREL:7FKF:WSY5
 Docker Root Dir: /usr/local/google/docker
 Debug Mode: true
  File Descriptors: 24
  Goroutines: 36
  System Time: 2021-02-12T13:40:50.695925257-08:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false
 
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: Support for cgroup v2 is experimental

afbjorklund · 2021-02-13T08:22:48Z

Basically they haven't configured their Docker installation "properly"

(alternatively you could say that their Linux vendor chose to disable some of the kernel capabilities for them, by default)

afbjorklund · 2021-02-18T17:07:54Z

Seems like it is now an error with Docker, like it was in Podman, so we should just avoid the flag. It was ignored, anyway...

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

medyagh · 2021-02-18T19:17:53Z

this PR ://github.com//pull/10507 didnt fully fix

we need to check for cat /sys/fs/cgroup/memory/memory.limit_in_bytes
to not add memory limits ( I will add a PR for that)

afbjorklund · 2021-02-18T20:18:30Z

Pretty sure I misunderstood the settings, as well. Not specifiying swap means that you want swap...

By default, the container can swap the same amount of assigned memory, which means that the overall hard limit would be around 256m when you set --memory 128m.

https://thorsten-hans.com/limit-memory-for-docker-containers

As far as I know, memory.limit_in_bytes is available by default and it was memswap that was toggled ?

But I don't have a system to reproduce this, worked OK on Debian Sid.

afbjorklund · 2021-02-18T20:26:47Z

The current checks were explicitly only to check for the boot parameter of Debian/Ubuntu:

https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

        // memory subsystem checks and adjustments
        if resources.Memory != 0 && resources.Memory < linuxMinMemory {
                return warnings, fmt.Errorf("Minimum memory limit allowed is 6MB")
        }
        if resources.Memory > 0 && !sysInfo.MemoryLimit {
                warnings = append(warnings, "Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.")
                resources.Memory = 0
                resources.MemorySwap = -1
        }
        if resources.Memory > 0 && resources.MemorySwap != -1 && !sysInfo.SwapLimit {
                warnings = append(warnings, "Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.")
                resources.MemorySwap = -1
        }
        if resources.Memory > 0 && resources.MemorySwap > 0 && resources.MemorySwap < resources.Memory {
                return warnings, fmt.Errorf("Minimum memoryswap limit should be larger than memory limit, see usage")
        }
        if resources.Memory == 0 && resources.MemorySwap > 0 && !update {
                return warnings, fmt.Errorf("You should always set the Memory limit when using Memoryswap limit, see usage")
        }

cgroups v1

        mountPoint, ok := cgMounts["memory"]
        if !ok {
                warnings = append(warnings, "Your kernel does not support cgroup memory limit")
                return warnings
        }
        info.MemoryLimit = ok

        info.SwapLimit = cgroupEnabled(mountPoint, "memory.memsw.limit_in_bytes")
        if !info.SwapLimit {
                warnings = append(warnings, "Your kernel does not support swap memory limit")
        }

cgroups v2

       if _, ok := controllers["memory"]; !ok {
                warnings = append(warnings, "Unable to find memory controller")
                return warnings
        }

        info.MemoryLimit = true
        info.SwapLimit = true

Not check for some weird system with /sys/fs/cgroup/memory but without /sys/fs/cgroup/memory/memory.limit_in_bytes

afbjorklund · 2021-02-18T20:29:18Z

Noticed a funky detail on podman, as well...

$ sudo podman run --memory 256m --memory-swap -1 busybox true
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15ee61e]

goroutine 1 [running]:
github.com/containers/podman/v2/cmd/podman/common.getMemoryLimits(0xc00035cc00, 0x27d8da0, 0x1b05550, 0x17, 0xc000594868)
	cmd/podman/common/specgen.go:153 +0x33e
github.com/containers/podman/v2/cmd/podman/common.FillOutSpecGen(0xc00035cc00, 0x27d8da0, 0xc0002b9f20, 0x2, 0x6, 0x0, 0x18ab660)
	cmd/podman/common/specgen.go:430 +0xda6
github.com/containers/podman/v2/cmd/podman/containers.run(0x2748b80, 0xc0002b9f20, 0x2, 0x6, 0x0, 0x0)
	cmd/podman/containers/run.go:171 +0x46f
github.com/spf13/cobra.(*Command).execute(0x2748b80, 0xc000138020, 0x6, 0x6, 0x2748b80, 0xc000138020)
	vendor/github.com/spf13/cobra/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x275c140, 0xc000130010, 0x18d6c80, 0x280b340)
	vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	vendor/github.com/spf13/cobra/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	vendor/github.com/spf13/cobra/command.go:888
main.Execute()
	cmd/podman/root.go:92 +0xec
main.main()
	cmd/podman/main.go:36 +0x92

💥

Reported as ~~containers/podman#9429~~ (fixed on master, but not 3.0.x)

afbjorklund · 2021-02-18T20:42:31Z

this PR #10507 didnt fully fix

@medya : it seems like you didn't merge #10468 ? so there was no check in place, for cgroups v2 ?

awan1 · 2021-03-31T00:08:26Z

I'm running into this issue again after my system OS updated, but I'm not sure what exactly was updated.

/etc/os-release says

NAME="Debian GNU/Linux"
ID=debian
VERSION_CODENAME=rodete
VERSION_ID=rodete

System Settings says

Cinnamon Version 4.8.6
Linux Kernel 5.7.17-1rodete5-amd64

Attempted: minikube delete followed by minikube start

Output:

ᐅ minikube start
😄  minikube v1.17.1 on Debian rodete
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=3900mb --memory-swap=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914.scope/memory.swap.max: permission denied: unknown.

Followed the post-install steps, which solved this problem for me before: https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities. Confirmed that /etc/default/grub contained GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1", ran sudo update-grub, rebooted machine. Did this twice, with same minikube start error message.

Happy to open new bug and/or provide new information, but this seemed like the same error message so I thought I'd reuse this bug.

afbjorklund · 2021-03-31T06:30:42Z

I think the latest Google Linux updates to cgroups v2, and there was a regression in how minikube handles that.

sharifelgamal · 2021-03-31T18:22:46Z

@awan1 I noticed you're using minikube 1.17.1, which is a full version old now. Can you try upgrading to v1.18.1 and see if that helps? It's working on my rodete machine.

costiser · 2021-03-31T19:38:09Z

Have you tried the fix indicated above?
https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

spowelljr · 2021-03-31T19:52:50Z

Also working on my machine as well on v1.18.1

Linux version 5.7.17-1rodete5-amd64

awan1 · 2021-03-31T22:16:57Z

@sharifelgamal great catch - updating to minikube v1.18.1 made minikube start work without doing anything else. Thank you! I'll add running minikube update-check to my debugging process.

medyagh changed the title ~~Minikube fails to start with docker driver - error: memory.swap.max: permission denied~~ Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" Feb 12, 2021

medyagh added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 12, 2021

medyagh self-assigned this Feb 12, 2021

medyagh added this to the v1.18.0 milestone Feb 12, 2021

medyagh removed their assignment Feb 12, 2021

afbjorklund mentioned this issue Feb 13, 2021

Make sure to show debian warning also for cgroup 2 #10468

Merged

medyagh self-assigned this Feb 16, 2021

afbjorklund mentioned this issue Feb 18, 2021

Failed to start minikube on Debian: memory.swap.max: permission denied: unknown. #10505

Closed

afbjorklund mentioned this issue Feb 18, 2021

Stop using --memory-swap if it is not available #10507

Merged

afbjorklund removed the priority/backlog Higher priority than priority/awaiting-more-evidence. label Feb 18, 2021

tstromberg mentioned this issue Feb 18, 2021

auto-setting 32 GB memory on Docker Driver ? #10459

Closed

medyagh closed this as completed in #10468 Feb 18, 2021

This was referenced Feb 18, 2021

Stop using --memory for cgroup v2. #10511

Closed

cgroup v2: skip setting --memory limits when not configurable. #10512

Merged

afbjorklund mentioned this issue Feb 20, 2021

improve error message for cgroup limit #10519

Closed

afbjorklund reopened this Feb 20, 2021

medyagh closed this as completed Feb 24, 2021

afbjorklund mentioned this issue Mar 27, 2021

docker says You should always set the Memory limit when using Memoryswap limit, see usage. #10935

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" #10371

Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" #10371

costiser commented Feb 5, 2021 •

edited

Loading

afbjorklund commented Feb 5, 2021 •

edited

Loading

costiser commented Feb 5, 2021

awan1 commented Feb 5, 2021 •

edited

Loading

afbjorklund commented Feb 5, 2021 •

edited

Loading

costiser commented Feb 6, 2021

medyagh commented Feb 12, 2021

afbjorklund commented Feb 13, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021

medyagh commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

awan1 commented Mar 31, 2021

afbjorklund commented Mar 31, 2021

sharifelgamal commented Mar 31, 2021

costiser commented Mar 31, 2021

spowelljr commented Mar 31, 2021

awan1 commented Mar 31, 2021

Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" #10371

Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied" #10371

Comments

costiser commented Feb 5, 2021 • edited Loading

afbjorklund commented Feb 5, 2021 • edited Loading

costiser commented Feb 5, 2021

awan1 commented Feb 5, 2021 • edited Loading

afbjorklund commented Feb 5, 2021 • edited Loading

costiser commented Feb 6, 2021

medyagh commented Feb 12, 2021

afbjorklund commented Feb 13, 2021 • edited Loading

afbjorklund commented Feb 18, 2021

medyagh commented Feb 18, 2021 • edited Loading

afbjorklund commented Feb 18, 2021 • edited Loading

afbjorklund commented Feb 18, 2021 • edited Loading

cgroups v1

cgroups v2

afbjorklund commented Feb 18, 2021 • edited Loading

afbjorklund commented Feb 18, 2021 • edited Loading

awan1 commented Mar 31, 2021

afbjorklund commented Mar 31, 2021

sharifelgamal commented Mar 31, 2021

costiser commented Mar 31, 2021

spowelljr commented Mar 31, 2021

awan1 commented Mar 31, 2021

costiser commented Feb 5, 2021 •

edited

Loading

afbjorklund commented Feb 5, 2021 •

edited

Loading

awan1 commented Feb 5, 2021 •

edited

Loading

afbjorklund commented Feb 5, 2021 •

edited

Loading

afbjorklund commented Feb 13, 2021 •

edited

Loading

medyagh commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading

afbjorklund commented Feb 18, 2021 •

edited

Loading