CGv1 + runc quadlet tests: Job for service failed because the control process exited with error code.

[cevich]

Issue Description

On systems using Cgroups v1 with runc, quadlet fails with a message similar to Error: mkdir /sys/fs/cgroup/pids/user.slice/user-2878.slice/[email protected]/runtime: permission denied

Steps to reproduce the issue

Steps to reproduce the issue

1. On a Debian SID or RHEL 8 VM
2. As a regular user
3. Run make localsystem

Describe the results you received

Something similar to the logs in CI:

[+1023s] not ok 325 quadlet - basic
         # (from function `assert' in file test/system/[helpers.bash, line 643](https://github.com/containers/podman/blob/1caf5ff8079a601bec77c6ed8292cabc61c60fdd/test/system/helpers.bash#L643),
         #  from function `service_setup' in file test/system/[252-quadlet.bats, line 87](https://github.com/containers/podman/blob/1caf5ff8079a601bec77c6ed8292cabc61c60fdd/test/system/252-quadlet.bats#L87),
         #  in test file test/system/[252-quadlet.bats, line 130](https://github.com/containers/podman/blob/1caf5ff8079a601bec77c6ed8292cabc61c60fdd/test/system/252-quadlet.bats#L130))
         #   `service_setup $QUADLET_SERVICE_NAME' failed
         # $ podman rm -t 0 --all --force --ignore
         # $ podman ps --all --external --format {{.ID}} {{.Names}}
         # $ podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
         # quay.io/libpod/testimage:20221018 f5a99120db64
         # # Automatically generated by /var/tmp/go/src/github.com/containers/podman/bin/quadlet
         # #
         # [X-Container]
         # Image=quay.io/libpod/testimage:20221018
         # Exec=sh -c "echo STARTED CONTAINER; echo "READY=1" | socat -u STDIN unix-sendto:$NOTIFY_SOCKET; top"
         # Notify=yes
         #
         # [Unit]
         # SourcePath=/tmp/podman_bats.ixl88m/quadlet.ofnKSG/basic_kX3MKPlUT1.container
         # RequiresMountsFor=%t/containers
         #
         # [Service]
         # Environment=PODMAN_SYSTEMD_UNIT=%n
         # KillMode=mixed
         # ExecStopPost=-/var/tmp/go/src/github.com/containers/podman/bin/podman rm -f -i --cidfile=%t/%N.cid
         # ExecStopPost=-rm -f %t/%N.cid
         # Delegate=yes
         # Type=notify
         # NotifyAccess=all
         # SyslogIdentifier=%N
         # ExecStart=/var/tmp/go/src/github.com/containers/podman/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --sdnotify=container -d quay.io/libpod/testimage:20221018 sh -c "echo STARTED CONTAINER; echo READY=1 | socat -u STDIN unix-sendto:$NOTIFY_SOCKET; top"
         # $ systemctl  start basic_kX3MKPlUT1.service
         # Job for basic_kX3MKPlUT1.service failed because the control process exited with error code.
         # See "systemctl --user status basic_kX3MKPlUT1.service" and "journalctl --user -xeu basic_kX3MKPlUT1.service" for details.
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #|     FAIL: Error starting systemd unit basic_kX3MKPlUT1.service
         # #| expected: -eq '0'
         # #|   actual:     '1'
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Describe the results you expected

All rootless quadlet tests should pass

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon_2.1.3+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: unknown'
  cpuUtilization:
    idlePercent: 59.59
    systemPercent: 13.55
    userPercent: 26.86
  cpus: 2
  distribution:
    codename: bookworm
    distribution: debian
    version: "12.03"
  eventLogger: journald
  hostname: cirrus-task-5606318455652352
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.0-3-cloud-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 3081449472
  memTotal: 4116254720
  networkBackend: netavark
  ociRuntime:
    name: runc
    package: runc_1.1.4+ds1-1+b1_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4+ds1
      commit: 1.1.4+ds1-1+b1
      spec: 1.0.2-dev
      go: go1.19.4
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 26m 33.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: mirror.gcr.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
  docker.io/library:
    Blocked: false
    Insecure: false
    Location: quay.io/libpod
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io/library
    PullFromMirror: ""
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 211116445696
  graphRootUsed: 4976545792
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0-dev
  Built: 1675874150
  BuiltTime: Wed Feb  8 16:35:50 2023
  GitCommit: 1caf5ff8079a601bec77c6ed8292cabc61c60fdd
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0-dev

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Debian GNU/Linux bookworm/sid \n \l

Kernel: 6.1.0-3-cloud-amd64
Cgroups: tmpfs
dpkg-query: no packages found matching containers-common
dpkg-query: no packages found matching cri-o-runc
conmon-2.1.3+ds1-1-amd64
containernetworking-plugins-1.1.1+ds1-3+b1-amd64
criu-3.17.1-2-amd64
crun-1.5+dfsg-1+b1-amd64
golang-2:1.19~1-amd64
libseccomp2-2.5.4-1+b3-amd64
podman-4.3.1+ds1-5+b1-amd64
runc-1.1.4+ds1-1+b1-amd64
skopeo-1.9.3+ds1-1-amd64
slirp4netns-1.2.0-1-amd64

Additional information

Ref: #17305 (comment)

[giuseppe]

we cannot use --cgroups=split on cgroup v1 as it requires write access to the cgroup. That is another test we need to skip on a cgroupv1 system

[cevich]

Thanks for the insight Giuseppe.

[cevich]

Just double-checked, we're correctly skipping the cgroups=split tests in containers/automation_images#250

[cevich]

Thanks @giuseppe