change in ulimit -u causes existing containers to not start

[Cydox]

Issue Description

Similar to #18555 but without using kube.

Updating my computer from Fedora Silverblue 38.20230526.0 to 38.20230527.0 slightly decreased the value output by ulimit -u from 63329 to 63304. This caused existing containers to not launch with the error:

crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

The containers never had a ulimit set manually on the command line, however looking at the output of podman inspect the ulimit value did get saved into the config causing a subsequent decrease in ulimit to break the containers.

Steps to reproduce the issue

Not a full reproduction as I'm not sure how to easily decrease the value of ulimit -u for your system, but just demonstrating that without using --ulimit a value does get saved into the config:

1. podman create test docker.io/alpine:3.18
2. podman inspect test

Describe the results you received

Output includes:

"Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 63329,
                         "Hard": 63329
                    }
               ],

Describe the results you expected

Output includes:

"Ulimits": [],

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 85.64
    systemPercent: 4.35
    userPercent: 10.01
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: silverblue
    version: "38"
  eventLogger: journald
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.2.15-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 12819206144
  memTotal: 16687968256
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 16687034368
  swapTotal: 16687034368
  uptime: 0h 10m 8.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/jan/.config/containers/storage.conf
  containerStore:
    number: 35
    paused: 0
    running: 1
    stopped: 34
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/jan/.local/share/containers/storage
  graphRootAllocated: 1004390125568
  graphRootUsed: 115150499840
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 43
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/jan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486942
  BuiltTime: Fri Apr 14 17:42:22 2023
  GitCommit: ""
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[Cydox]

Some updates. Couldn't reproduce this issue running podman inside a container. So that made my debugging a little more difficult on Silverblue (had to resort to some print debugging ...).

Thus far I have tracked the issue down to this line:

podman/pkg/specgen/generate/oci.go

Line 24 in e7dc507

if s.Rlimits == nil {

In the container (where the issue does not occur), len(s.Rlimits) is 0 and s.Rlimits is nil. On the host (where the issue occurs) len(s.Rlimits) is also 0 but s.Rlimits is not nil.

Changing this line to:

if len(s.Rlimits) == 0 {

fixes the issue.

Still investigating if something preceding this is the root-cause.

[Cydox]

The above fix does not work for the rootfull setup on my host.

[Cydox]

I've identified some more things in the ulimit handling that I think needs some cleaning up. WIP fix is at: https://github.com/Cydox/podman/tree/fix-ulimit

[Luap99]

This seems to be a duplicate of #18696