Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exec: crun: setns(pid=87793, CLONE_SOMETHING): Operation not permitted: OCI permission denied #1264

Closed
edsantiago opened this issue May 4, 2023 · 11 comments · Fixed by #1265
Closed

exec: crun: setns(pid=87793, CLONE_SOMETHING): Operation not permitted: OCI permission denied #1264

edsantiago opened this issue May 4, 2023 · 11 comments · Fixed by #1265

Comments

@edsantiago
Copy link
Member 

$ podman [options] exec test id
Error: crun: setns(pid=87793, CLONE_NEWUSER): Operation not permitted: OCI permission denied

and

# podman-remote [options] exec ctr-name cat /etc/passwd
Error: crun: setns(pid=177787, CLONE_*): Operation not permitted: OCI permission denied
...
Error: read unixpacket @->/proc/self/fd/13/attach: read: connection reset by peer (presumably because remote)

Only two, both in my no-flake-retries PR. root & rootless, local & remote, f38 and rawhide. Different tests, the common factor is that both tests involve userns.

  • May 1 : rawhide rootless, in podman create --userns=keep-id + podman exec - correct names of user and group
  • May 4 : f38 remote root, in [It] podman run --userns=keep-id can add users
@Luap99
Copy link
Member

Luap99 commented May 5, 2023

I am pretty sure the first one was a variant of containers/podman#17903. New userns so it was no longer able to join the existing user namespace in exec.

The second one is something different.

@github-actions
Copy link

github-actions bot commented Jun 5, 2023

A friendly reminder that this issue had no activity for 30 days.

@edsantiago
Copy link
Member Author

Here it is again, today, f37 remote root:

# podman-remote [options] exec ctr-name cat /etc/passwd
Error: crun: setns(pid=244604, CLONE_*): Operation not permitted: OCI permission denied
time="2023-06-08T13:17:19-05:00" level=error msg="attaching to container 4ca3f95a2bf7209ce4105a2a64d4e099cdb3fa9bfb55ef630685c1e459154aa6 exec session 19f05156c610a9fd09341d8959f73c4ddc63d4fde194bb50f523fb138ccbb777: crun: setns(pid=244604, CLONE_*): Operation not permitted: OCI permission denied"
Error: read unixpacket @->/proc/self/fd/13/attach: read: connection reset by peer

@Luap99
Copy link
Member

Luap99 commented Jun 9, 2023

@giuseppe PTAL, AFAIK userns=keep-id as root is just a NOP so it should be no different then a normal podman run without userns.

@giuseppe
Copy link
Member

giuseppe commented Jun 9, 2023

with root we still create a new user namespace but we map the entire IDs space.

@edsantiago
Copy link
Member Author

Another one, f37 root remote. @giuseppe thank you for your answer, do you think you could code up a fix please?

@edsantiago edsantiago mentioned this issue Jun 20, 2023
Closed
@github-actions
Copy link

github-actions bot commented Aug 4, 2023

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Aug 5, 2023

@giuseppe were you able to work on this one?

@giuseppe
Copy link
Member

giuseppe commented Aug 7, 2023

I'll take another look, last time I looked at this issue I have not managed to reproduce locally

@edsantiago
Copy link
Member Author

Here's the full list:

  • fedora-37 : int remote fedora-37 root host sqlite [remote]
    • 06-14-2023 18:52 in Podman UserNS support podman run --userns=keep-id can add users
    • 06-08-2023 14:32 in Podman UserNS support podman run --userns=keep-id can add users
  • fedora-38 : int podman fedora-38 root host sqlite
    • 06-21-2023 09:17 in Podman UserNS support podman run --userns=keep-id can add users
  • fedora-38 : int remote fedora-38 root host boltdb [remote]
  • rawhide : int podman rawhide root host sqlite
    • 06-20-2023 13:30 in Podman UserNS support podman run --userns=keep-id can add users
  • rawhide : int remote rawhide root host sqlite [remote]
    • 06-26-2023 21:50 in Podman UserNS support podman run --userns=keep-id can add users

All of them in containers/podman#17831, because the flake seems to pass on retry. I have not been hammering on that PR as much in the past month, which is one reason why we haven't seen it in the last month.

@giuseppe giuseppe transferred this issue from containers/podman Aug 8, 2023
giuseppe added a commit to giuseppe/crun that referenced this issue Aug 8, 2023
@giuseppe
linux: simplify setns with pidfd
d13934a 
I can't recall why the original code was setting the user namespace
separately, probably it was an issue specific to an old kernel.

I've tried with the current kernel and it works fine to do all the
setns() in one shoot so let's simplify the code and avoid an EPERM
error on the second setns() when using a user namespace.

Closes: containers#1264

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/crun that referenced this issue Aug 8, 2023
@giuseppe
linux: simplify setns with pidfd
ea1ba0d 
I can't recall why the original code was setting the user namespace
separately, probably it was an issue specific to an old kernel.

I've tried with the current kernel and it works fine to do all the
setns() in one shoot so let's simplify the code and avoid an EPERM
error on the second setns() when using a user namespace.

This caused an issue when the PID 1 didn't execve yet the container
process, and joining the user namespace first would cause the user to
look access to the container PID 1 namespaces.

Closes: containers#1264

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Aug 8, 2023
Merged
@giuseppe
Copy link
Member

giuseppe commented Aug 8, 2023

opened a PR: #1265

giuseppe added a commit to giuseppe/crun that referenced this issue Aug 8, 2023
@giuseppe
linux: simplify setns with pidfd
88441d9 
This caused an issue when the PID 1 didn't execve yet the container
process, and joining the user namespace first would cause the user to
look access to the container PID 1 namespaces.

When running with enough permissions, setns() will allow to join all
the namespaces in one shoot, so let's use it.

On any error, attempt the fallback mechanism to join each namespace
separately.

Closes: containers#1264

Signed-off-by: Giuseppe Scrivano <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

linux: simplify setns with pidfd giuseppe/crun
4 participants
@giuseppe @edsantiago @rhatdan @Luap99