Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error #6857

mybigman · 2020-07-06T02:45:08Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

unable to create containers

Steps to reproduce the issue:

podman run -it --rm --name test alpine:latest

Describe the results you received:
Error: setrlimit RLIMIT_NOFILE: Invalid argument: OCI runtime error

Describe the results you expected:
The container to start

Additional information you deem important (e.g. issue happens only occasionally):
podman run --ulimit host -it --rm --name test alpine:latest
Using --ulimit host solves the issue.

Output of podman version:

Version:      2.0.1
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 10:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 128
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: hostname.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.6-201.fc32.x86_64
  linkmode: dynamic
  memFree: 268024225792
  memTotal: 270138871808
  ociRuntime:
    name: crun
    package: crun-0.14-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14
      commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 34359734272
  swapTotal: 34359734272
  uptime: 27m 59.31s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.1-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.1-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical Machine

ulimit -u --soft
8388608

ulimit -u --hard
8388608

The text was updated successfully, but these errors were encountered:

vrothberg · 2020-07-06T08:49:41Z

Thanks for opening the issue, @mybigman! We're working to fix the issue. Cc @giuseppe @mheon @rhatdan

mheon · 2020-07-06T14:39:31Z

This looks sufficiently different from the PID limit that I'm not sure the proposed fix for that will fix this

mheon · 2020-07-06T14:40:25Z

It's also rather unusual that you're seeing this with NOFILE, and not the pids limit.

It looks like it is set to a non-standard value here...

rhatdan · 2020-07-06T18:24:02Z

Could you show the podman inspect $CTR and look at the ulimit settings?

Do you have a containers.conf on your system with some settings?

mybigman · 2020-07-07T01:14:25Z

@rhatdan I searched the system and there at two containers.conf

~/.config/containers/containers.conf < I did not create this file and renaming this file made no difference.

default_ulimits = [
  "nproc=200000:400000",
]

/usr/share/containers/containers.conf < I have not made any changes to this file and everything appears to be commented out.

I can only do an inspect on a container that's ran with --ulimit host since I can no longer create containers without it or an existing container that was built prior to the issue.

built with --ulimit host

"Ulimits": [],

existing container

 "Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 524288,
        "Hard": 524288
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

rhatdan · 2020-07-07T17:51:03Z

If you remove the file from your homedir, you still can not create a container?

mybigman · 2020-07-08T04:02:50Z

Correct... without this file I still cant create the container.

rhatdan · 2020-07-08T10:30:48Z

If you create the container, with the create command and then inspect it you are seeing ulimits in the config?

podman --version

mybigman · 2020-07-08T10:53:35Z

Doing podman create alpine:latest ls has the following in the ulimits.

"Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 524288,
        "Hard": 1024
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

Podman version is in first post.

podman version 2.0.1

rhatdan · 2020-07-08T10:59:29Z

Could you download and try https://koji.fedoraproject.org/koji/buildinfo?buildID=1537752

rhatdan · 2020-07-08T11:01:45Z

It should shortly be in updates-testing.
https://bodhi.fedoraproject.org/updates/FEDORA-2020-19924b556e

# dnf -y update --enablerepo=updates-testing podman

Sadly it was not there when I just tried.

mybigman · 2020-07-09T02:03:35Z

Installed the 2.0.2 version from testing and works as expected. Thanks for persevering.

ulimits

 "Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 1024,
        "Hard": 524288
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

rugk · 2020-07-15T12:04:07Z

This issue is still happening in Podman v2.0.2, although it was supposed to be fixed there.

For more information see:
containers/toolbox#500 (comment)

HarryMichal · 2020-07-15T13:31:41Z

I also hit this issue on Fedora Silverblue Rawhide. I encounter it mainly when I try to enter a toolbox container created with an older version of Podman (pre v2.0.2). New steps to reproduce the issue:

podman run --ulimit=nofile=524288:524288 --ulimit=nproc=63173:63173 alpine echo Hello World

One of the --ulimit options can be dropped, I included them because they used to be set both.

The ^ command fails with:

Error: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error

Output of rpm -q podman:

podman-2.1.0-0.70.dev.git3d33590.fc33.x86_64

Output of podman version:

Version:      2.1.0-dev
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.0-dev
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-0.5.dev.giteff699e.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19-dev, commit: 3c47d3797172bffa8ab02661ac4805b593cfb4ba'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: file
  hostname: harry-ntb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.0-0.rc5.1.fc33.x86_64
  linkmode: dynamic
  memFree: 2233049088
  memTotal: 16632356864
  ociRuntime:
    name: crun
    package: crun-0.14-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14
      commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-2.dev.git4c6befe.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: 4c6befe05c3137232cf06a5c2879daf4c20be6b1
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 8401186816
  swapTotal: 8401186816
  uptime: 20h 6m 17.41s (Approximately 0.83 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/harry/.config/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 3
    stopped: 11
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.0-6.dev.git50ab2c2.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.2
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.2
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/harry/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 12
  runRoot: /run/user/1000
  volumePath: /var/home/harry/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.1.0-dev

mheon · 2020-07-15T14:06:50Z

Is this Silverblue?

mheon · 2020-07-15T14:07:07Z

And can you include a full podman info?

HarryMichal · 2020-07-15T14:34:10Z

I updated my previous comment with the needed info.

mheon · 2020-07-20T15:32:33Z

This has always been broken - I went back to v1.9.3 and that command is equally nonfunctional. This does not seem like a Podman issue. We briefly made containers with bad rlimits, and those containers were and still are unusable. New containers will work properly.

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 6, 2020

rhatdan closed this as completed Jul 9, 2020

HarryMichal mentioned this issue Jul 14, 2020

"Error: failed to start container fedora-toolbox-32" on Fedora Silverblue containers/toolbox#500

Closed

HarryMichal mentioned this issue Sep 22, 2020

Failed to start container containers/toolbox#557

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error #6857

Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error #6857

mybigman commented Jul 6, 2020

vrothberg commented Jul 6, 2020

mheon commented Jul 6, 2020

mheon commented Jul 6, 2020

rhatdan commented Jul 6, 2020

mybigman commented Jul 7, 2020

rhatdan commented Jul 7, 2020

mybigman commented Jul 8, 2020

rhatdan commented Jul 8, 2020

mybigman commented Jul 8, 2020

rhatdan commented Jul 8, 2020

rhatdan commented Jul 8, 2020

mybigman commented Jul 9, 2020

rugk commented Jul 15, 2020 •

edited

Loading

HarryMichal commented Jul 15, 2020 •

edited

Loading

mheon commented Jul 15, 2020

mheon commented Jul 15, 2020

HarryMichal commented Jul 15, 2020

mheon commented Jul 20, 2020

Error: setrlimit RLIMIT_NOFILE: Invalid argument: OCI runtime error #6857

Error: setrlimit RLIMIT_NOFILE: Invalid argument: OCI runtime error #6857

Comments

mybigman commented Jul 6, 2020

vrothberg commented Jul 6, 2020

mheon commented Jul 6, 2020

mheon commented Jul 6, 2020

rhatdan commented Jul 6, 2020

mybigman commented Jul 7, 2020

rhatdan commented Jul 7, 2020

mybigman commented Jul 8, 2020

rhatdan commented Jul 8, 2020

mybigman commented Jul 8, 2020

rhatdan commented Jul 8, 2020

rhatdan commented Jul 8, 2020

mybigman commented Jul 9, 2020

rugk commented Jul 15, 2020 • edited Loading

HarryMichal commented Jul 15, 2020 • edited Loading

mheon commented Jul 15, 2020

mheon commented Jul 15, 2020

HarryMichal commented Jul 15, 2020

mheon commented Jul 20, 2020

Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error #6857

Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error #6857

rugk commented Jul 15, 2020 •

edited

Loading

HarryMichal commented Jul 15, 2020 •

edited

Loading