podman(1) start | run failing: ERRO[0008] "container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: no cgroup mount found in mountinfo: OCI runtime error"

[nmvega]

Hello Friends:

Today I rebuilt a docker image from scratch (via a Dockerfile(5), which I've been iteratively doing (for weeks) to get the image perfect. That docker image is built to support ROOTLESS podman(1) within it, and that has worked perfectly until after today's rebuild.

Just a reminder from this GitHub issue that my setup is like this (which works). I only mention it because I did a dnf -y update in layers (1) and (2) this
weekend, and of course also in layer (3) when rebuilding the docker image today:

• (1) Bare-metal host: FC-33 x86_64
  • (2) Linux / LXC container: FC-33 x86_64
    • (3) Docker-CE container: FC-33 x86_64
      • (4) Podman

Please see below. Any ideas? Was a new version of podman(1) released?

Here is the script, taking note of my in-line comment where failure is triggered:

#! /usr/bin/bash

podman pod create \
    --name cassandra \
    --network acme \
    --share net \
    --publish 9042:9042

podman create \
    --pod cassandra \
    --name cassandra_container \
    --hostname cassandra \
    --restart always \
    --add-host cassandra:127.0.0.1 \
    docker.io/library/cassandra:latest

podman start cassandra_container  # <-- Failure is triggered by this.

Here's the error thrown by the start command:

jdoe@fedora33$ podman start cassandra_container
ERRO[0008] error starting some container dependencies   
ERRO[0008] "container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: no cgroup mount found in mountinfo: OCI runtime error" 
Error: unable to start container "2bfb0320ffe6e2ad4a95c236e14a105a83d102c19b2d2ebf6c359dc537f706e7": error starting some containers: internal libpod error

Diagnostic information:

podman(1) veraion:

jdoe@fedora33$ podman version
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.15.8
Built:        Thu Apr 22 13:21:33 2021
OS/Arch:      linux/amd64

./containers.conf:

jdoe@fedora33$ cat ~/.config/containers/containers.conf 
[engine]
events_logger = "file"
cgroup_manager = "cgroupfs"

./storage.conf:

jdoe@fedora33$ cat ~/.config/containers/storage.conf
[storage]
driver = "overlay"
graphroot             = "$HOME/.local/share/containers/storage"
rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
additionalimagestores = []
mount_program = "/usr/bin/fuse-overlayfs"

[storage.options.overlay]
mountopt = "nodev"
mount_program = "/usr/bin/fuse-overlayfs"

[storage.options.thinpool]

[nmvega]

Additional information (a running comment):

jdoe@fedora33$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

jdoe@fedora33$ grep cgroup /proc/self/mountinfo
1210 1200 0:94 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,inode64

jdoe@fedora33$ podman info
host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: file
  hostname: 8e0f6b133dc6
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.15-200.fc33.x86_64
  linkmode: dynamic
  memFree: 38025424896
  memTotal: 67374006272
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /home/jdoe/Downloads/TEMP.d/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 34h 53m 41.94s (Approximately 1.42 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/jdoe/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev
  graphRoot: /home/jdoe/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /home/jdoe/Downloads/TEMP.d/containers-user-1000/containers
  volumePath: /home/jdoe/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1619097693
  BuiltTime: Thu Apr 22 13:21:33 2021
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.1.2

[nmvega]

UPDATE and QUESTIONS (thank you in advance):

In reviewing this GitHub issue, I solved this issue by dnf() installing crun(1), which my Dockerfile(5) doesn't install (but will install going forward).

QUESTIONS:

1. What happened in my Docker rebuild & Fedora O/S updates that things now require crun(1)? A new podman(1) version installled?

2. I noticed this in the above podman(1) info output: cgroupVersion: v1. Should I be using v2? This isn't enterprise. If yes, where do I specify this?

PS: I read this informative and well-written article by you guys, which helped me a lot.

If you can answer the two above questions (and have any other insight), I'd appreciate it. ☺️

Thank you!

[mheon]

ERRO[0008] "container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: no cgroup mount found in mountinfo: OCI runtime error" 

Alright, never seen this one before. Potentially a runc bug? Swapping to crun may be avoiding said bug. We made crun default in 3.0 but still support runc if crun is unavailable.

[mheon]

Cgroups v1 vs v2 is an OS-level configuration option. We definitely prefer v2. You need to add systemd.unified_cgroup_hierarchy=1 to your Kernel command line (You probably have this already there, but set to 0, given Fedora now defaults to v2).

[nmvega]

Cgroups v1 vs v2 is an OS-level configuration option. We definitely prefer v2. You need to add systemd.unified_cgroup_hierarchy=1 to your Kernel command line (You probably have this already there, but set to 0, given Fedora now defaults to v2).

@mheon Thank you.

• Do I need to do this at every layer in my nesting? I'm guessing the change should be at the bare-metal host, since everything below that is inherited from there, but let me know which levels (thanky).
• And for Fedora33, what file is the easiest place to set this? In /etc/default/grub(5), by appending to the GRUB_CMDLINE_LINUX= directive?

[nmvega]

Cgroups v1 vs v2 is an OS-level configuration option. We definitely prefer v2. You need to add systemd.unified_cgroup_hierarchy=1 to your Kernel command line (You probably have this already there, but set to 0, given Fedora now defaults to v2).

@mheon PS: You may be correct that I sidestepped this issue by installing crun(1), so your hypothesis here still lingers. =:)

[mheon]

@nmvega It does need to be done on the host, I think that everything underneath probably picks it up from there, though if there's a completely separate systemd in play anywhere (Maybe the LXC container? Maybe?) that might also need to be set to use v2.

[nmvega]

@mheon Yea, that makes sense (bare-metal + possibly the nested LXC containers, too). I'll try cgroups v2 somewhat soon. I have four levels of nesting for my little project so I'm cautious, buy I'll try it once I have the project stabilized. =:) Thanks for your help.

[rhatdan]

I am lost, is this still an issue? Is this a problem with runc?

[nmvega]

@rhatdan Well yes and no. My installation of crun sidestepped (masked) the original issue, which arises while using runc -- and may still indeed be an issue to investigate. That's why I didn't close this issue, in case you wanted to follow-up with respect to runc.

But I'm all good here as long as I use crun. So check out the OP (the opening comment). That may still be a problem because of runc or something else that happens while using runc.

You can close this issue for now if you like. Thanks for the assistance!

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Ok I am going to close this issue.