Cannot limit cpu with kube file: Invalid Argument

[0xb35c]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When I try to podman play kube an existing kube file with cpu-limits, it gives me an error saying:

error starting container f488fc03d8949bb4fd92ffbccadf6fda6b326322288587171a419e2985c60352: writing file `cpu.max`: Invalid argument: OCI runtime error

On the other hand, using the --cpu flag with podman run seems to work - at least it doesn't print any errors.

Steps to reproduce the issue:
Basically use or create an kube file, which worked before and add cpu limits. Running these lines works for me:

podman pod create -n test
podman run --pod test -d --cpus 1 docker.artifactory.schmid.local/alpine sleep infinity
podman generate kube test > test.yml
podman pod rm -f test
podman play kube test.yml

Alternatively this is the kube file podman created:

# Generation of Kubernetes YAML is still under development!
#
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-3.3.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2021-09-30T11:24:04Z"
  labels:
    app: test
  name: test
spec:
  containers:
  - command:
    - sleep
    - infinity
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: container
      value: podman
    - name: http_proxy
      value: http://localhost:3128
    - name: https_proxy
      value: http://localhost:3128
    image: alpine:latest
    name: hungrycarver
    resources:
      limits:
        cpu: "1"
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
    workingDir: /
  dnsConfig: {}
  restartPolicy: Never
status: {}

Describe the results you received:

Pod:
715f1d870df69292e2802a1a8a72e7f9846be4544e54d1508c161460273e0a5c
Container:
f488fc03d8949bb4fd92ffbccadf6fda6b326322288587171a419e2985c60352

error starting container f488fc03d8949bb4fd92ffbccadf6fda6b326322288587171a419e2985c60352: writing file `cpu.max`: Invalid argument: OCI runtime error

Error: failed to start 1 containers

Describe the results you expected:

Pod:
715f1d870df69292e2802a1a8a72e7f9846be4544e54d1508c161460273e0a5c
Container:
f488fc03d8949bb4fd92ffbccadf6fda6b326322288587171a419e2985c60352

Additional information you deem important (e.g. issue happens only occasionally):
I was having the issue from point 26 from the troubleshooting guide, but I think this is resolved:

> cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"
cpu io memory pids

I also was trying to use runc as runtime - similar error:

> podman play kube test.yml
Pod:
5ee58b6c4c9e7a2d2a24429e4a43fc8c8fbba610b62d838a0af1f4844d303770
Container:
a9832d225f760e03c20bdb6d5670367e7435cb2238e6b3ac0b2aad2fa4d8cffc

error starting container a9832d225f760e03c20bdb6d5670367e7435cb2238e6b3ac0b2aad2fa4d8cffc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "100 100000": write /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/user.slice/user-libpod_pod_5ee58b6c4c9e7a2d2a24429e4a43fc8c8fbba610b62d838a0af1f4844d303770.slice/libpod-a9832d225f760e03c20bdb6d5670367e7435cb2238e6b3ac0b2aad2fa4d8cffc.scope/cpu.max: invalid argument: OCI runtime error

Output of podman version:

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Git Commit:   4c5283fabff2de5145838f1847a5a7b2b1fbc0a5-dirty
Built:        Wed Sep  1 19:27:46 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.30-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: 2792c16f4436f1887a7070d9ad99d9c29742f38a'
  cpus: 4
  distribution:
    distribution: endeavouros
    version: unknown
  eventLogger: journald
  hostname: phoenix
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65537
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65537
  kernel: 5.14.8-arch1-1
  linkmode: dynamic
  memFree: 4011237376
  memTotal: 8304197632
  ociRuntime:
    name: /usr/bin/crun
    package: /usr/bin/crun is owned by crun 1.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.1
      commit: 5b341a145c4f515f96f55e3e7760d1c79ec3cf1f
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 22h 12m 51.25s (Approximately 0.92 days)
registries:
  search:
  - registry-1.docker.io
store:
  configFile: /home/schwendb/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 2
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.7.1-1
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/schwendb/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 10
  runRoot: /run/user/1000/containers
  volumePath: /home/schwendb/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630517266
  BuiltTime: Wed Sep  1 19:27:46 2021
  GitCommit: 4c5283fabff2de5145838f1847a5a7b2b1fbc0a5-dirty
  GoVersion: go1.17
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

> yay -Q podman
podman 3.3.1-1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

[mheon]

Rootless Cgroups v2 with systemd driver - it looks like limits should work.

This is probably the CPU controller not being mounted - question is why podman run works. Maybe it discards the limit?

@giuseppe PTAL

[giuseppe]

PR here: #11806

[tyler92]

@giuseppe In release v3.4.7 the following settings in kube yaml work:

resources:
  limits:
    # CPU limit, 1m is 0.1%
    cpu: 333m

But since v4.0.0 it doesn't work. And it looks like #11806 broke it. Am I wrong?