Error: readlink: Permission denied: OCI permission denied

[edsantiago]

This is one of those flakes that appear at random, in random tests, making it almost impossible for my flake-logger to isolate. Symptom is:

Running: podman [options] run [...]  quay.io/libpod/alpine:latest [command]
Error: readlink: Permission denied: OCI permission denied

One interesting thing: I see this (sometimes, not always) with run -d, implying that the error happens before detach.

Podman UserNS support [It] podman --userns=container:CTR

• fedora-34 : int podman fedora-34 rootless host
  • PR Always spawn a cleanup process with exec #10405
    • 06-11 09:42

Podman pod create [It] podman pod container can override pod not sharing pid

• fedora-33 : int podman fedora-33 rootless host
  • PR rootless: fix fast join userns path #10611
    • 06-09 10:19
• fedora-34 : int podman fedora-34 rootless host
  • PR Always spawn a cleanup process with exec #10405
    • 06-10 15:02
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 11:23

Podman run [It] podman shared PID NS container share SELinux labels

• fedora-33 : int podman fedora-33 rootless host
  • PR container: ignore named hierarchies #10609
    • 06-10 10:43
• fedora-34 : int podman fedora-34 rootless host
  • PR Fix pre-checkpointing #10639
    • 06-10 11:01

Podman pod create [It] podman pod correctly sets up PIDNS

• fedora-34 : int podman fedora-34 rootless host
  • PR fix: uid/gid for volume mounted to existing dir #10905
    • 07-12 09:47
  • PR Fix network connect race with docker-compose #10654
    • 06-11 10:46
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR Add support for podman remote build -f - . #10651
    • 06-11 07:14

Podman top [It] podman pod top on pod with containers in same pid namespace

• fedora-33 : int podman fedora-33 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 08:28
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 11:23

sys: podman selinux: shared context in (some) namespaces

• fedora-34 : sys podman fedora-34 rootless host
  • PR Makefile: remove install.cni #10803
    • 06-29 07:10

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Podman top [It] podman pod top on pod with containers in same pid namespace

• fedora-33 : int podman fedora-33 rootless host
  • PR skip flaking auto-update test #11176
    • 08-10 06:00

Podman pod create [It] podman pod container can override pod not sharing pid

• fedora-34 : int podman fedora-34 rootless host
  • PR podman-remote build use .containerignore over .dockerignore  #10913
    • 07-14 08:58

Podman pod create [It] podman pod correctly sets up PIDNS

• fedora-34 : int podman fedora-34 rootless host
  • PR fix: uid/gid for volume mounted to existing dir #10905
    • 07-12 09:47

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

I've been keeping flake logs for a few months, and on August 17 I wrote a tool to grep those logs. The last occurrence of "readlink" on the same line as "Permission denied" was August 10 (already logged in comment above). It's possible that the error message changed, also possible that the issue magically fixed itself. I don't think there's anything I can do aside from close the issue.

[edsantiago]

Reopening: this is still happening.

Podman pod create [It] podman pod create with --userns=keep-id can add users

• fedora-34 : int podman fedora-34 rootless host
  • PR Exclude already built sources for static build #12235
    • 11-09 04:36
• fedora-35 : int podman fedora-35 rootless host
  • PR e2e tests: use HaveKey() and HaveLen() when possible #12482
    • 12-02 10:46

Podman pod create [It] podman pod create with --userns=auto

• fedora-35 : int podman fedora-35 rootless host
  • PR Fix possible rootless netns cleanup race #12469
    • 12-01 14:04
    • 12-01 14:04
    • 12-01 14:04

Podman pod create [It] podman pod create with --userns=keep-id

• ubuntu-2110 : int podman ubuntu-2110 rootless host
  • PR Cirrus: Update to Ubuntu 21.10 + Disable F33 #12256
    • 11-11 14:59

Podman pod create [It] podman pod create --userns=auto:size=%d

• fedora-34 : int podman fedora-34 rootless host
  • PR test/system: podman run with log-opt option #12221
    • 11-08 09:51

Two observations:

• There's an awful lot of userns in the list above. Much more than I think can be explained by coincidence.
• In logs where this happens, it seems to happen often, in other tests.

[vrothberg]

          podman pod create --userns=auto:size=%d
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:739
         
         [BeforeEach] Podman pod create
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:28
         [It] podman pod create --userns=auto:size=%d
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:739
         Running: podman [options] pod create --userns=auto:size=500 --name testPod
         6092a09b3c751d0a768f5f7193034c2d793fbd3680fe7c42e344b8dcaf202fd4
         Running: podman [options] run --pod testPod quay.io/libpod/alpine:latest cat /proc/self/uid_map
                  0          1        500
         Running: podman [options] pod create --userns=auto:size=3000 --name testPod-1
         40ec609fac65ff52559ee8033d475074eb009f924d909ed11a42ab3b458a6bb9
         Running: podman [options] run --pod testPod-1 quay.io/libpod/alpine:latest cat /proc/self/uid_map
         Error: readlink: Permission denied: OCI permission denied

@giuseppe, could this be a crun issue?

[rhatdan]

No you are running a container within a Pod and already have set the userns size to 500, now you are attempting to set a container in the same user namesapce to use 3000 uids, which is not allowed.

What should happen here is --pod and userns option should conflict if container is sharing userns with the pod.

[rhatdan]

BTW We should do that for all of the namespace options. If you share a namespace with the pod, then you should use the options set within the pod.

[rhatdan]

593d090 just did this for conflicting --hostname when in a Pod.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

# ./bin/podman pod create --userns=auto:500  --name dan
Error: invalid option specified: "500"
# ./bin/podman pod create --userns=auto:size=500  --name dan
9af8b13ebbb413e871ba1cfabfeda3d09342988b2500dabec5394b7a095acdf0
# ./bin/podman create --userns=auto:size=3000 --pod dan fedora echo hi
Error: --userns and --pod cannot be set together

Looks like current code at least conflicts setting USERNS within a Pod that is using user namespace.

Not sure if this is enough to close down this issue.

[edsantiago]

Most recent instance was Jan 12:

[sys] 236 podman selinux: shared context in (some) namespaces

• fedora-34 : sys podman fedora-34 rootless host
  • PR [v3.4.2-rhel] buildah bud tests: skip failing tests #12832
    • 01-12 14:04

That instance does not seem to have anything to do with userns or pods:

[+0933s] [not ok 236 podman selinux: shared context in (some) namespaces]()
         # (from function `die' in file test/system/helpers.bash, line 448,
         #  from function `run_podman' in file test/system/helpers.bash, line 221,
         #  in test file test/system/410-selinux.bats, line 125)
         #   `run_podman run --rm --pid container:myctr $IMAGE cat -v /proc/self/attr/current' failed with status 126
         # $ podman rm --all --force
         # $ podman ps --all --external --format {{.ID}} {{.Names}}
         # $ podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
         # quay.io/libpod/testimage:20210610 9f9ec7f2fdef
         # $ podman run -d --name myctr quay.io/libpod/testimage:20210610 top
         # ac8e1d127b201330e12858aa66306362b24359e4bbf31b3f8f42283e97536b2f
         # $ podman exec myctr cat -v /proc/self/attr/current
         # system_u:system_r:container_t:s0:c68,c413^@
         # $ podman run --name myctr2 --ipc container:myctr quay.io/libpod/testimage:20210610 cat -v /proc/self/attr/current
         # system_u:system_r:container_t:s0:c68,c413^@
         # $ podman run --rm --pid container:myctr quay.io/libpod/testimage:20210610 cat -v /proc/self/attr/current
         # Error: readlink: Permission denied: OCI permission denied
         # [ rc=126 (** EXPECTED 0 **) ]

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Nothing in my logs since Jan 12; make of that what you will.

[rhatdan]

3 Months, Time to close...

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago Close?

[edsantiago]

No new instances since Jan 12. Okay, closing.

[edsantiago]

Seen today in f37 rootless. Reopening.

  podman pod top on pod with containers in same pid namespace
...
$ podman [options] pod create --infra=false --share 
d8b01eac6d1ea48c3f663d4b63b3fc54ceef58e1922331c5aeda1c963141dcd4
$ podman [options] run -d --pod d8b01eac6d1ea48c3f663d4b63b3fc54ceef58e1922331c5aeda1c963141dcd4 quay.io/libpod/alpine:latest top -d 2
6e812b0d7d847c624707baafadfd1d128ac5bcedc18b0704683e8829422e7d23
$ podman [options] run -d --pod d8b01eac6d1ea48c3f663d4b63b3fc54ceef58e1922331c5aeda1c963141dcd4 --pid container:6e812b0d7d847c624707baafadfd1d128ac5bcedc18b0704683e8829422e7d23 quay.io/libpod/alpine:latest top -d 2
d4fe59cf384803bd5340d32240722a8601b588ca99d9da6fbae5756b74f53b53
$ podman [options] pod top d8b01eac6d1ea48c3f663d4b63b3fc54ceef58e1922331c5aeda1c963141dcd4
Error: error extracting PID namespace: readlink /proc/59714/ns/pid: permission denied

[vrothberg]

@giuseppe could this be a race in psgo?

[rhatdan]

@edsantiago waiting for response from you

[edsantiago]

No instances since the one on May 1