Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd" #6084

Closed
avikivity opened this issue May 5, 2020 · 92 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@avikivity
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. podman run -it --rm fedora:32

Describe the results you received:

Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

Describe the results you expected:

#

Additional information you deem important (e.g. issue happens only occasionally):

Happens all the time

Output of podman version:

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.1
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: tmp.scylladb.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.7-300.fc32.x86_64
  memFree: 5275238400
  memTotal: 33541488640
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 16869486592
  swapTotal: 16869486592
  uptime: 93h 3m 6.55s (Approximately 3.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/avi/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/avi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/avi/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.9.1-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Fully updated Fedora 32.

@avikivity
Copy link
Author

Update: this started to happen on my second Fedora 32 machine. On the other hand, I rebooted the first one (due to an unrelated problem :( ) and the problem went away.

I'm running Linux 5.6.7 on the still-broken machine, so it's unlikely to be kernel version related.

@mheon
Copy link
Member

mheon commented May 5, 2020

Can you provide ~/.config/containers/libpod.conf from a system that reproduces?

@avikivity
Copy link
Author

That one has no ~/.config/containers/libpod.conf.

$ cat ~/.config/containers/libpod.conf
cat: /home/avi/.config/containers/libpod.conf: No such file or directory
$ podman run -it --rm fedora:32
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

@avikivity
Copy link
Author

Linux avi 5.6.7-300.fc32.x86_64 #1 SMP Thu Apr 23 14:13:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

@mheon
Copy link
Member

mheon commented May 5, 2020

It's definitely a v2 system from podman info, so the issue must be the cgroup manager. Can you try a container with --cgroup-manager=systemd and see if that works?

@mheon
Copy link
Member

mheon commented May 5, 2020

@rhatdan Does this look like a potential containers.conf issue to you? Maybe a default resource limit specified in the config?

@hiisukun
Copy link

hiisukun commented May 6, 2020

I also bumped into this. First time running podman on Fedora 32, trying to launch a couple of containers.

Output:

[hiisukun@serv:~]$ podman run -it --rm fedora:32
Trying to pull registry.fedoraproject.org/fedora:32...
Getting image source signatures
Copying blob 3088721d7dbf done  
Copying config d81c91deec done  
Writing manifest to image destination
Storing signatures
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

podman version:

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.1
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: serv
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.8-300.fc32.x86_64
  memFree: 3510607872
  memTotal: 14608658432
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 7417098240
  swapTotal: 7423913984
  uptime: 50h 46m 55.42s (Approximately 2.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/hiisukun/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/hiisukun/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/hiisukun/.local/share/containers/storage/volumes

rpm -q podman:

podman-1.9.1-1.fc32.x86_64

uname -a:

Linux omni 5.6.8-300.fc32.x86_64 #1 SMP Wed Apr 29 19:01:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

libpod.conf:

[hiisukun@serv:~]$ cat ~/.config/containers/libpod.conf
cat: /home/hiisukun/.config/containers/libpod.conf: No such file or directory

Other:

  • /usr/share/containers/libpod.conf is whatever came with my package (unmodified by me)
  • Specifying --cgroup-manager=systemd on the command line didn't help
  • Everything is commented out in /usr/share/containers/containers.conf that is not a section heading
  • The container I started with was docker.io/linuxserver/unifi-controller, and it gives the same error as fedora:32.

@hiisukun
Copy link

hiisukun commented May 6, 2020

Well, after shutting down the various things and rebooting, it has indeed gone away as per @avikivity mentioning. Hopefully the source of the problem can be fixed since rebooting isn't always convenient.

@avikivity
Copy link
Author

It's definitely a v2 system from podman info, so the issue must be the cgroup manager. Can you try a container with --cgroup-manager=systemd and see if that works?

Tried it, no change.

@avikivity
Copy link
Author

Since I needed the machine to work I rebooted it (unfortunately, couldn't check if logout is sufficient since Fedora 32 loses the keyboard/mouse after logout). The problem went away on that machine too. Unfortunately this means I can't help with debugging it until it recurs again (and then I will be limited by needing podman to work).

@MartinNowak
Copy link

For me it was an outdated ~/.config/containers/libpod.conf config file that still had a cgroup_manager = "cgroupfs" line.
I don't remember having created that config file, so I suspect it was automatically created by podman (or maybe by me following some tutorial).
I removed that file and now running a container works without the --cgroup-manager=systemd override.
This doesn't sound exactly like OPs problem, but seems related enough to post here.

@mheon
Copy link
Member

mheon commented May 6, 2020

The reboot part of this seems bizarre...

If anyone else encounters this problem: can you try running podman system reset and see if things go back to normal after that?

@trebor8x
Copy link

trebor8x commented May 6, 2020

I am facing the same issue after upgrading Fedora 31 to 32.
Only config I could find is located at /usr/share/containers/libpod.conf and it states cgroup_manager = "systemd"

Running with debug output yields:

podman run --log-level=debug --rm -it debian bash
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs
DEBU[0000] Ignoring lipod.conf EventsLogger setting "journald". Use containers.conf if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] container-default [] private [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL  ...
...
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
...

Running podman system reset does not change the result.

Previously I removed the directory .local/share/containers/, but the error persists.

@mheon
Copy link
Member

mheon commented May 6, 2020

Can you provide full output for the command with debugging enabled?

Can you try manually setting --log-driver=k8s-file and seeing if that resolves the issue?

Finally, does systemctl --user as the user running Podman work?

@trebor8x
Copy link

trebor8x commented May 6, 2020

The command systemctl --user reports: Failed to connect to bus.
It is also impossible to start a new gnome-terminal by now.

@mheon
Copy link
Member

mheon commented May 6, 2020

Alright, I don't think this is Podman.
It sounds like the systemd user session is just not running, but starts after a reboot? Tempted to say it's a systemd bug?

@denesb
Copy link

denesb commented May 7, 2020

I'm also seeing this on Fedora 31.

@avikivity
Copy link
Author

@mheon please change the error message to "could not connect to dbus" then. The current error message looks like it's trying to help, but it's actively misleading.

@mheon
Copy link
Member

mheon commented May 7, 2020

Digging deeper, that part looks like a bug. We should be discarding resource limits and not erroring unless manually set, but it looks like containers.conf is forcing resources to be set on every Podman invocation.

@rhatdan From a brief parse, I'm pretty sure containers.conf is unconditionally setting the PID limit, which is causing us to blow up on cgroups v2 systems that are not using the systemd cgroup manager.

@amarshall
Copy link

I see the original error as well when using cgroup_manager="cgroupfs" (which I need to use due to a different error which I’ll likely open an issue for soon). Downgrading podman from 1.9.1 to 1.8.2 appears to resolve it.

@numas
Copy link

numas commented May 12, 2020

I experienced this too on my Fedora 31 laptop (been through 28 > 29 > 30 > 31 upgrades). An inelegant deletion of ~/.config/containers fixed it, but not sure if there are consequences in this.

@rhatdan
Copy link
Member

rhatdan commented May 12, 2020

Most likely you are fine. Podman will just use the defaults if that directory was removed. If you made any customizations, then there could be issues.

@Navdeepuniyal
Copy link

Navdeepuniyal commented May 14, 2020

I am facing the same issue when using REST API:

curl --header "Content-Type: application/json"   --request POST   --data '{
>   "image": "nginx",
>   "privileged": true,
>   "publish_image_ports": true,
>   "name": "rem-nginx"
> }' http://localhost:4041/containers/create

Error:

{"cause":"invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd","message":"CreateContainerFromCreateConfig(): invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd","response":500}

Any hints?

@rhatdan
Copy link
Member

rhatdan commented May 14, 2020

Did you remove the libpod.conf, and could you try out podman-1.9.2 in updates-testing?

@Navdeepuniyal
Copy link

Interestingly, it works with the restart.

@andrewgdunn
Copy link

andrewgdunn commented May 14, 2020

Seeing the same issue on two fresh F32 instances, rolling back podman-1.8.2 (whats currently "new" in F31) was our current solution. We'll be watching this ticket with intent!

@rhatdan
Copy link
Member

rhatdan commented May 14, 2020

I would love to see if people with this issue, see if it is fixed by podman 1.9.2

@mheon
Copy link
Member

mheon commented Jun 25, 2020

I don't believe any are necessary; we'll detect that /sys/fs/cgroup is using v2 when Podman is invoked, and begin using it.

@DevinBayly
Copy link

hmm, now I'm stumped. So if I don't need to modify the configuration how do I make podman start using cgroups v2? Does that mean I should only have to pass the correct flag to --cgroup-manager= when I do podman run with memory limits?

@mheon
Copy link
Member

mheon commented Jun 25, 2020

The system itself needs to be switched to Cgroups v2 - it should be a kernel parameter, though I don't know what the exact parameter is for Ubuntu.

@DevinBayly
Copy link

And just to make sure, I know I have cgroups v2 because of the result of grep cgroup /proc/filesystems which shows

nodev   cgroup
nodev   cgroup2

but I still need to find a way to switch the system to cgroups v2? It's not enough to have it already installed? Sorry this stuff is pretty foreign to me.

@mheon
Copy link
Member

mheon commented Jun 25, 2020

We only support cgroups v2 in unified mode - that is, only cgroup2 will be mounted.

@DevinBayly
Copy link

DevinBayly commented Jun 25, 2020

ok managed to follow these instructions https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter, and I tucked in the parameter systemd.unified_cgroup_hierarchy=1 and now podman info actually contains a line that says cgroupVersion: v2, but when i run podman run -it --rm -m=2g --memory-swap=-1 --cgroup-manager=systemd alpine /bin/bash
I get a new error
Error:sd-bus add match: Operation not permitted: OCI runtime permission denied error

Is this something else that folks have dealt with?

@mheon
Copy link
Member

mheon commented Jun 25, 2020

@giuseppe Any thoughts here?

@giuseppe
Copy link
Member

what version of systemd are you using? The error is coming from crun while trying to create the cgroup.

What is the value of the DBUS_SESSION_BUS_ADDRESS env variable?

@DevinBayly
Copy link

Thanks for helping out. The version of systemd is 237, and the value of the variable is DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus

@giuseppe
Copy link
Member

it seems like the systemd version is too old.

To verify it, try the command unshare -r systemd-run --user echo it works as rootless. Does it succeed?

@DevinBayly
Copy link

yes that command fails
Failed to start transient service unit: Access denied
So I should try to update systemd? Or perhaps that's only possible by upgrading to a newer version of Ubuntu?

@giuseppe
Copy link
Member

yes, you either need to upgrade systemd or use the cgroupfs backend for cgroups.

What happens is that we are not able to create a systemd scope as rootless while being in a user namespace.

@DevinBayly
Copy link

gotcha, I think for the moment I'll switch to a centos 8 machine and see if the same problems arise. But just to try to follow this thread to completion, I tried using --cgroup-manager=cgroupfs and I still got the error that is this issue's title. I've searched a bit but haven't immediately found instructions for setting up cgroupfs, are there specific steps I need to take before I can expect podman to work with cgroupfs? Let me know if we've reached the point where what I'm asking isn't specific to the issue here..

@skorhone
Copy link

Please re-open #6798, if you find this unrelated. I closed it for now

@DevinBayly
Copy link

I ran into the same issue with a fresh install of Centos 8 even after modifying the kernel parameters so that cgroups v2 was in use for podman. I feel like I'm still not understanding how to make use of the cgroupfs in the cli or if there's supposed to be a config file that I update. The only one I could find was the ~/.config/containers/storage.conf. Either way I still got
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd
when i ran podman run -it --rm -m=2g --memory-swap=-1 --cgroup-manager=systemd alpine
I'm going to try centos stream 8 to see if things are new enough to support rootless memory management for containers.

@RheaAyase
Copy link

Still very much present with podman-2.0.1-1.fc32
So I updated and restarted the whole spaceship which somehow ignored podman so then I updated podman and now the same error, back to square one, waiting for the next server restart in several months I guess.

@mheon
Copy link
Member

mheon commented Jul 2, 2020

Can you provide the error message you're seeing? Also, are you on Fedora or FCOS?

@RheaAyase
Copy link

The one in the $subject - after updating from 1.9.3 to 2.0.1

Restart solved it (came to the (obvious) realisation that the recent hardware upgrade and the removal of that supermicro bloat actually makes restarts without upgrades pretty fast.)

It did not however resolve everything, I still had to run all the systemd things with: XDG_RUNTIME_DIR=/run/user/$UID systemctl --user which is the earlier mentioned systemd issue that was discussed.

@mheon
Copy link
Member

mheon commented Jul 2, 2020

That's really wierd, because the code path that leads to that error should be completely disabled in v2.0.0 and up...

@RheaAyase
Copy link

Another interesting thing, perhaps coincidence, is that virsh no longer works from system services either. Had to change that to a --user as well.

@github-actions
Copy link

github-actions bot commented Aug 3, 2020

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Aug 3, 2020

We had some fixes for this in podman 2.0 releases. Is this fixed now, can we close this issue?

@andrewgdunn
Copy link

Personally I'm waiting for 2.1.x (stuck on 1.8.2 due to #7016). I'd not encountered this when I was doing 2.x testing though.

@avikivity
Copy link
Author

Original reporter here, haven't seen it in a long time. On 2.0.3 now.

@rhatdan rhatdan closed this as completed Aug 4, 2020
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Development

No branches or pull requests