Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rawhide (crun): rootless(?): podman exec broken: OCI runtime error #3937

Closed
edsantiago opened this issue Sep 4, 2019 · 9 comments · Fixed by #3959
Closed

rawhide (crun): rootless(?): podman exec broken: OCI runtime error #3937

edsantiago opened this issue Sep 4, 2019 · 9 comments · Fixed by #3959
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@edsantiago
Copy link
Member

On current rawhide with crun:

$ podman run -d --name foo alpine sleep 60
8c0855798e92d718be496e428c66a4e702ec4921575f006a5848fe53e41d50b8
$ podman exec foo true
Error: non zero exit code: 1: OCI runtime error
$ echo $?
1

I can't check root because podman-root is 100% broken (rhbz1748214).

With --log-level=debug:

...
DEBU[0000] Successfully started exec session 3985743e00dad6f2710b08790fb2ea75f96572b520aa2c16dc296cfb927e2856 in container c5875947634e0bf1f5804b19f361e7123b253f523cf43e0494ad04450d508419
[conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
ERRO[0000] non zero exit code: 1: OCI runtime error

podman-1.5.2-0.46.dev.git1d8a940.fc32.x86_64
crun-0.7-2.fc31.x86_64
package runc is not installed

@giuseppe
Copy link
Member

giuseppe commented Sep 5, 2019

it should work well with crun 0.8. The new version has different fixes for interacting with Podman and reporting the correct exit code.

@rhatdan
Copy link
Member

rhatdan commented Sep 5, 2019

@edsantiago can you test it out, reopen if this does not fix the issue.

@rhatdan rhatdan closed this as completed Sep 5, 2019
@edsantiago edsantiago reopened this Sep 5, 2019
@edsantiago
Copy link
Member Author

Problem persists with crun-0.8-1.fc32.x86_64 although the error is different:

$ podman exec foo true
Error: writing file '/sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-61c0251e87cd6312ce7e6176f2ad66f3260b675a079efa5d71ad79ed22e06671.scope/cgroup.procs': Permission denied: OCI runtime permission denied error

@rhatdan
Copy link
Member

rhatdan commented Sep 5, 2019

Progress...

@giuseppe
Copy link
Member

giuseppe commented Sep 5, 2019

it seems to fail moving a process from cgroup A to a cgroup B, where A is the current cgroup (owned by root) and B is the destination cgroup in the container.

Can you confirm the current cgroup is owned by root with `ls -l /sys/fs/cgroup/$(sed -e 's|0::||' /proc/self/cgroup)?

Does it work if you use something like systemd-run --user --scope podman exec foo echo hello ?

Not sure what would be the best way to handle it, we'd need to enforce somehow to run in a cgroup that is owned by the rootless user.

@giuseppe
Copy link
Member

giuseppe commented Sep 6, 2019

opened a PR to automatically create the scope: #3959

@edsantiago
Copy link
Member Author

Sorry for the late response. In case it's still helpful:

Can you confirm the current cgroup is owned by root with `ls -l /sys/fs/cgroup/$(sed -e 's|0::||' /proc/self/cgroup)?

$ (I took the liberty of adding `-d` to the command above)
drwxr-xr-x. 2 root root 0 Sep  9 15:15 /sys/fs/cgroup//user.slice/user-0.slice/session-18.scope
(ls -l, no -d, shows everything owned by root)

Does it work if you use something like systemd-run --user --scope podman exec foo echo hello ?

$ systemd-run --user --scope podman exec foo echo hello
Failed to create bus connection: No such file or directory
$ XDG_RUNTIME_DIR=/run/user/1000 !!
Running scope as unit: run-r51595d4cfd9a41db8c2d73c407893090.scope
hello

@juhp
Copy link
Contributor

juhp commented Sep 10, 2019

I hit this too now while trying to test toolbox in Rawhide.

giuseppe added a commit to giuseppe/libpod that referenced this issue Sep 12, 2019

Verified

This commit was signed with the committer’s verified signature.
giuseppe Giuseppe Scrivano
when running in rootless mode and using systemd as cgroup manager
create automatically a systemd scope when the user doesn't own the
current cgroup.

This solves a couple of issues:

on cgroup v2 it is necessary that a process before it can moved to a
different cgroup tree must be in a directory owned by the unprivileged
user.  This is not always true, e.g. when creating a session with su
-l.

Closes: containers#3937

Also, for running systemd in a container it was before necessary to
specify "systemd-run --scope --user podman ...", now this is done
automatically as part of this PR.

Signed-off-by: Giuseppe Scrivano <[email protected]>
@popsUlfr
Copy link

I think I encountered this problem on podman 1.9.0 in rootless mode :

When I try to exec :

$ podman --log-level debug exec -ti d3bfcb240aed bash
DEBU[0000] Attaching to container d3bfcb240aed2e55863dbff58af373b80a284777dbc93eadab336a58a120f7a6 exec session 53430ac13aad60bfe302e5b1b6d583ed5c1276538cdbd605ba9f4c2d3dbac03d 
DEBU[0000] connecting to socket /run/user/974/libpod/tmp/socket/53430ac13aad60bfe302e5b1b6d583ed5c1276538cdbd605ba9f4c2d3dbac03d/attach 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

                                                                          DEBU[0000] Received: 0                                  
DEBU[0000] Received: -1                                 
ERRO[0000] [conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
writing file `/sys/fs/cgroup//user.slice/user-974.slice/[email protected]/user.slice/libpod-d3bfcb240aed2e55863dbff58af373b80a284777dbc93eadab336a58a120f7a6.scope/cgroup.procs`: Permission denied: OCI runtime permission denied error

But if I prefix with systemd-run :

$ systemd-run --user --scope podman exec -ti d3bfcb240aed bash

It works.

Not sure if it is related but I have a dedicated podmanuser user on the system, so I SSH to my ssh user and then I sudo machinectl shell podmanuser@ to become the other user and get a systemd user session.

Here's my podman info :

host:
  arch: arm64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 1bddbf7051a973f4a4fecf06faa0c48e82f1e9e1'
  cpus: 4
  distribution:
    distribution: archarm
    version: unknown
  eventLogger: journald
  hostname: rock64
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 974
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
    uidmap:
    - container_id: 0
      host_id: 974
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
  kernel: 5.6.7-1-ARCH
  memFree: 3494100992
  memTotal: 4097687552
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 58m 41.45s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /var/lib/podmanuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /var/lib/podmanuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/974/containers
  volumePath: /var/lib/podmanuser/.local/share/containers/storage/volumes

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants