podman pod --create needs to support the --userns=keep-id flag.

[rlpowell]

/kind bug

Description

The following used to work before I upgraded to Fedora 32:

$ podman pod create -n hello-pod
$ podman run --pod hello-pod --userns=keep-id --name hello-cont hello-world
Error: open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI runtime permission denied error

This seems to be very similar to #7766 , and as such may be more of a Fedora issue than a podman issue, but I don't understand the relevant systems well enough to be sure.

And, same as #7766, it is fixable by commenting out:

default_sysctls = [
 "net.ipv4.ping_group_range=0 1",
]

in my /usr/share/containers/containers.conf, which I had not previously modified.

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 24
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: lebna
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-300.fc33.x86_64
  linkmode: dynamic
  memFree: 87874744320
  memTotal: 101080154112
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.4-4.dev.giteecccdb.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 1h 30m 41.54s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/rlpowell/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/rlpowell/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/rlpowell/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Wed Oct  7 09:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-12.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Bare metal, F32

[rlpowell]

Summoning @rhatdan , since he handled the seems-to-be-directly-related issue.

[rhatdan]

If you change your default to be 0 0 rather then 0 1, this should fix the problem.

We have updates going on in Fedora 33 and updated containers/common with the new default.

[carsonbaker]

Fwiw, changing to 0 0 doesn't fix the issue for me. I have to comment out the configuration line.

[rlpowell]

As with carsonbaker, this "0 0" does not solve it for me:

rlpowell@jukni> sudo grep ping_gr /usr/share/containers/containers.conf
# for example:"net.ipv4.ping_group_range = 0 1000".
    "net.ipv4.ping_group_range=0 0",
rlpowell@jukni> podman pod kill hello-pod ; podman pod rm hello-pod  ; podman pod create -n hello-pod ; podman kill hello-cont ; podman rm hello-cont ; podman run --pod hello-pod --userns=keep-id --name hello-cont hello-world
998696b7bff0324fc9623cb236b31b74d2979e0173cd49f1ddf022e9df0f5bb7
998696b7bff0324fc9623cb236b31b74d2979e0173cd49f1ddf022e9df0f5bb7
3f4199ceccb705ae1bd058f28954c5bd942e8449a757154396e53b5ee7e79b36
Error: no container with name or ID hello-cont found: no such container
Error: no container with name or ID hello-cont found: no such container
Error: open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI runtime permission denied error

vs.

rlpowell@jukni> sudo grep ping_gr /usr/share/containers/containers.conf
# for example:"net.ipv4.ping_group_range = 0 1000".
#     "net.ipv4.ping_group_range=0 0",
rlpowell@jukni> podman pod kill hello-pod ; podman pod rm hello-pod  ; podman pod create -n hello-pod ; podman kill hello-cont ; podman rm hello-cont ; podman run --pod hello-pod --userns=keep-id --name hello-cont hello-world
3f4199ceccb705ae1bd058f28954c5bd942e8449a757154396e53b5ee7e79b36
3f4199ceccb705ae1bd058f28954c5bd942e8449a757154396e53b5ee7e79b36
9868fbdfcfa5033cc205055baaad3d383dd2333a964292e9e486d5da41ee8dd8
Error: no container with name or ID hello-cont found: no such container
Error: no container with name or ID hello-cont found: no such container

Hello from Docker!
This message shows that your installation appears to be working correctly.

[rlpowell]

I don't seem to have perms to re-open; hopefully someone else does?

[rhatdan]

This seems to be distribution based issue. If you distribution/kernel does not supporting setting this sysctl, then it should be removed. We are shipping this by default in Fedora and it is set as the default in containers/common in the config file, but if you remove the config file, it will not be set.

What distro are you seeing this on? How old is the kernel?

[rlpowell]

Fedora 33, just updated like last week, 5.8.18-300.fc33.x86_64

[rhatdan]

Ok this is a pod/container issue. If I run the container outside of a pod, it works, but when it is run within a container, it fails.
The issue seems to be joining the namespace of a existing container. Without the setting.

[rhatdan]

@mheon This seems like it might be a serious bug. Basically with current settings pods will not work.

[rhatdan]

Ok, I have dug deeper into this, the issue is that we are running the infra container and the container in two different user namespaces. The pod is running in the default usernamespace while the container is running in --userns=keep-id. We need to run the pod in the keep-id user namespace for this to work.

[rhatdan]

If you remve the --userns=keep-id, does it work for you?

[rlpowell]

I'm sorry, yes, I should have said this at the beginning, but tickling this bug requires both pods and --userns=keep-id; drop either the --pod or the --userns and it's fine.

Unfortunately, my automation specifically uses that exact combination :). --userns is so that I can have a database container and a web container with everything running with only the permissions and ID of the calling service account, and --pod is so that the web container can see the DB container without exposing it anywhere else.

On the --userns front, if there's a better way to achieve "mount this data directory owned by my user into this rootless container without having the files end up with weird owner IDs", I'm more than happy to listen. Like, if there's a way to map things so that the "apache" UID in the container matches the calling user's UID outside of the container, that'd be great, but it's not obvious to me how to do that with --uidmap.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rlpowell]

I disagree that this is stale.

[skovmand]

We also have the same issue.

[rhatdan]

I agree, I have started a PR to fix this, but I have not had time to finish it, I really could use someone else to take it over.

#8393
@umohnani8 Do you think you could take this over?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[developedsoftware]

Just stumbled into this exact issue. Is there a work around until it is fixed ? Been googling like a mad man !

[rhatdan]

Sorry no solution yet.

[dzintars]

Looking for this as well

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[pciavald]

Hope i'm not hijacking the thread, i have a similar issue with the error

Error: mount `mqueue` to `/dev/mqueue`: Operation not permitted: OCI permission denied

when using --userns=keep-id on a container inside a pod, on up-to-date debian 10 (raspios) kernel 5.10.17-v7+ (armv7l) with podman 3.0.1.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rlpowell]

I disagree that this is stale, in the sense that as far as I know it's still a problem.

[umohnani8]

Yup, it is not stale. I have a PR in progress #10589

[umohnani8]

Fixed in #10589