Docker fails to start additional containers with cgroup memory allocation error

[JensInc]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

• At least 10 or 20 container can be started even in virtual environments.

Actual behavior

• got error message below when trying to start any additional container

ERROR: for service_name  Cannot start service service_name: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/memory/docker/0b7ea1bef4d4638aedbf984a1caaf9b53f742ad7affeb028f9a8b6ca91bc5f01: permission denied: unknown

• error cannot removed by restarting docker daemon
• error disappears (it is possible to start at least 4 docker containers more) when restarting the virtual machine but appears again when restarting some container
• similar to Docker fails to start containers with cgroup memory allocation error. #841

Steps to reproduce the behavior

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:38 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:50 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 49
  Running: 8
  Paused: 0
  Stopped: 41
 Images: 142
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.15.0
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 32GiB
 Name: xxx.stratoserver.net
 ID: TUBV:CATO:OMC3:V3YI:2IAY:AOR4:KNVK:JLJ6:U7BW:V53O:SVFT:KPNR
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.)

• running kernel 4.15 (Ubuntu 18.04)
• running in virtual environment (Ubuntu 20.04 at Strato)
• running docker 20.10.7
• provider allows 1300 threads
• fraction to be used by non operating system: processes: systemctl show --property=DefaultTasksMax : DefaultTasksMax=550 (default setting had been increased at advised by hoster support)
• no zombie processes (ps aux | grep 'Z')
• docker stats reports ~ 120 threads
• htop report ~330 thread by docker (+20 by other processes)
• /etc/docker/daemon.json

{
    "iptables": false
}
{
    "dns": ["8.8.8.8","85.214.7.22", "81.169.163.106", "81.169.148.34", "10.91.3.31", "10.90.3.31", "10.90.7.14"]
}
{
     "userland-proxy": false
}