podman-in-docker: building image fails on cgroupv2-enabled systems

[stac47]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Building image with podman from a docker container fails on recent OS (ubuntu 21.10) but works on old systems (Red Hat Enterprise Linux Server release 7.9 (Maipo)). It is probably linked to the fact cgroupv2 is enabled on not.

Steps to reproduce the issue:

1. Using the podman image described in the article https://www.redhat.com/sysadmin/podman-inside-container, build a basic image as described in the following oneliner:

% docker run -i --privileged quay.io/podman/stable podman build - <<EOF
FROM fedora
RUN echo hello
EOF
STEP 1/2: FROM fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying config sha256:3059bef432ebb91a6a51d8f5cf20b033041dbddb3cab79628c1eb3412cbde0ae
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo hello
error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo hello]: writing file `/sys/fs/cgroup/buildah-buildah957585528/cgroup.procs`: Operation not supported
: exit status 1
Error: error building at STEP "RUN echo hello": error while running runtime: exit status 1
zsh: exit 125   docker run -i --privileged quay.io/podman/stable podman build - <<<''

Describe the results you received:

The image build fails when on the RUN directive:

error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo hello]: writing file `/sys/fs/cgroup/buildah-buildah957585528/cgroup.procs`: Operation not supported

Describe the results you expected:

Running the same on Red Hat Enterprise Linux Server release 7.9 (Maipo)

% docker run -i --privileged quay.io/podman/stable podman build - <<EOF
heredoc> FROM fedora
heredoc> RUN echo hello
heredoc> EOF
Unable to find image 'quay.io/podman/stable:latest' locally
latest: Pulling from podman/stable
4545346f2a49: Already exists
b59d2d8f6c51: Pull complete
6f9c9697d233: Pull complete
64c213ae2a2d: Pull complete
efc31865fe95: Pull complete
f59ce0390c36: Pull complete
6176dbbe111e: Pull complete
6f4545bc34ef: Pull complete
Digest: sha256:c9758557a650a5b71f9aa440c3ba853bfb248f63cb583d60f9014ae408fd5d6f
Status: Downloaded newer image for quay.io/podman/stable:latest
STEP 1/2: FROM fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying config sha256:3059bef432ebb91a6a51d8f5cf20b033041dbddb3cab79628c1eb3412cbde0ae
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo hello
hello
COMMIT
--> c326f21e77a
c326f21e77a1e2ba6bcdf88fce26875a14d92a42bce40f3b432d0987febe5482

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

% docker run -i --privileged quay.io/podman/stable podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Fri Nov 12 20:25:37 2021
OS/Arch:      linux/amd64

I also tried with podman 3.4.3 coming when we run dnf update --refresh --enablerepo=updates-testing podman in fedora:latest.

Output of podman info --debug:

% docker run -i --privileged quay.io/podman/stable podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: aa3ea5642c31
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.0-21-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1977335808
  memTotal: 16779907072
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 577h 16m 44.04s (Approximately 24.04 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 20:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

For comparison with the system on which it works:

% docker run -i --privileged quay.io/podman/stable podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - memory
  - devices
  - freezer
  - net_cls
  - blkio
  - perf_event
  - hugetlb
  - pids
  - net_prio
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: bc9e523cb51b
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 3.10.0-1160.45.1.el7.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1610510336
  memTotal: 16806350848
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 705789952
  swapTotal: 1073737728
  uptime: 441h 47m 29.01s (Approximately 18.38 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 20:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

% docker run -i --privileged quay.io/podman/stable rpm -q podman
podman-3.4.2-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

@giuseppe PTAL

[giuseppe]

I think it might happen because the Podman process is at the cgroup root inside the parent container, so the OCI fails to create a new cgroup and move a process there. I am not sure if it is something Podman needs to deal with though.

To confirm what I said above, could you please try running these commands first in the Docker container before running Podman?

# mkdir /sys/fs/cgroup/init
# cat /sys/fs/cgroup/cgroup.procs | tee /sys/fs/cgroup/init/cgroup.procs

[vrothberg]

@giuseppe I can confirm that this is resolving the issue.

[stac47]

Unfortunately, I cannot confirm it works. I am stuck here:

% docker run -it --privileged quay.io/podman/stable bash
[root@59dcb74c5a73 /]#  mkdir /sys/fs/cgroup/init
[root@59dcb74c5a73 /]# cat /sys/fs/cgroup/cgroup.procs
1
22
[root@59dcb74c5a73 /]# cat /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs
cat: write error: Invalid argument
[root@59dcb74c5a73 /]#  cat /sys/fs/cgroup/cgroup.procs | tee /sys/fs/cgroup/init/cgroup.procs
1
24
25
tee: /sys/fs/cgroup/init/cgroup.procs: Invalid argument

I don't know why I cannot write /sys/fs/cgroup/init/cgroup.procs

[root@59dcb74c5a73 /]# mount | grep cgroup
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Tell me if I missed something.

[vrothberg]

I can reproduce the error as well; it doesn't seem to work on my machine anymore.

[dtimm]

Since cgroupsv2 doesn't allow a process to be in multiple subgroups, you can't write a pid to /sys/fs/cgroup/init/cgroup.procs that already exist in /sys/fs/cgroup/cgroup.procs.

This should work, for instance:

sleep 60 &
echo $! > /sys/fs/cgroup/init/cgroup.procs

[giuseppe]

If you write the pid to a different cgroup, the process is migrated there.

On cgroup v2 you must be careful though that the destination cgroup does not have controllers enabled for its sub cgroups

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@giuseppe @stac47 @vrothberg What should we do with this issue, does not look like something Podman can fix. But @stac47 question was never answered. Since this seems like more of a discussion at this point I am transferring.

[vrothberg]

Looks like a crun issue to me. It works with runc.

@giuseppe can you take another look?

[pravinrajr9]

how to make it work with runc?

[giuseppe]

it doesn't depend on crun or runc.

It depends on Podman running in the root cgroup. You need to create a subcgroup and move Podman there.

[pravinrajr9]

Can you please provide the steps. By the way issue is same in docker as well docker in docker.

[AkihiroSuda]

Is the host Docker rootless?
Providing docker info and docker version output might be helpful

[sonman]

Running the Docker host with --cgroupns=host may help.