Many podman commands simply print "cannot set user namespace"

[MLNW]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Earlier today I tried to build an existing Dockerfile but upgrading it to use Ubuntu 21.04 instead of 20.10. This failed rather quickly because of the glibc issue. Therefore I tried to upgrade podman from 2.2.1 to the 3.1.x version available through the Kubic project (I am on Centos8). This sadly did not fix the issue.

Since I wasn't able to build 21.04 I reverted back to 20.10 and rebuilt the image. While doing that the WSL2 instance this is all running in filled up all available hard disk space, which actually crashed the instance. To fix this I cleared out all images build with podman and buildah to recover ~30 GBs of disk space. Afterward I shrunk the image with a guide.

I used the following commands to clear all images:

podman system prune
podman system prune --all
podman system prune --all --force
podman rmi $(podman images -qa)
podman rmi --force
buildah rm --all

Now after clearing out all images and resizing the disk I am no longer able to use podman as most commands simply print:

> podman version
cannot set user namespace

I searched the internet for this string and I cannot find anything. Do you have any clue how to fix this? I already tried reinstalling the old version of podman but I cannot use it productively anymore.

Output of podman version:

> podman version
cannot set user namespace
> sudo podman version
Version:      2.2.1
API Version:  2
Go Version:   go1.14.12
Built:        Mon Feb 22 05:51:35 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

❯ podman info --debug
cannot set user namespace

Package info (e.g. output of rpm -q podman or apt list podman):

❯ rpm -q podman
podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

• WSL2
• CentOS Linux release 8.3.2011

[mheon]

That error string does not occur in the code anywhere I can find. The WSL bit does make me suspicious, given that creating a new user namespace is a kernel operation.

[rhatdan]

newuidman?

[rhatdan]

I can not find that string in shadow-utils, crun, runc, podman conmon.

@giuseppe Ideas?

[giuseppe]

the error is coming from the join_namespace_or_die function in pkg/rootless/rootless_linux.c.

Is there any file left in $XDG_RUNTIME_DIR or any podman process running?

[MLNW]

I restarted the WSL2 instance by shutting down the WSL core. That did not help.

I don't have an environment variable set called $XDG_RUNTIME_DIR. There also does not seem to be any other podman process running.

[giuseppe]

I am not familiar with WSL2.

Is there a /run/user/$UID directory? Is that on tmpfs?

[MLNW]

No, it does not seem that way

❯ ll /run/user/$UID
ls: cannot access '/run/user/1000': No such file or directory

[giuseppe]

can you find out where podman stores its rundir? Is it under /tmp?

[MLNW]

Seems like that is correct

❯ ll /tmp | grep run
drwx------  7 lucas lucas  4.0K Apr 26 10:09 podman-run-1000
drwx------  3 lucas lucas  4.0K Mar  9 10:57 run-1000

[istyf]

At last! I have Googled this so many times now trying to find the cause for this issue. I also managed to fill my WSL2 and ran a podman prune to recover disk space. Now more or less all podman commands give me "cannot set user namespace". Just updated to podman 3.1.2.

[istyf]

Since this seems to be the same issue I am offering up my info as well:

sudo podman version
Version: 3.1.2
API Version: 3.1.2
Go Version: go1.15.2
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64

sudo podman info --debug
host:
arch: amd64
buildahVersion: 1.20.1
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.27, commit: '
cpus: 12
distribution:
distribution: ubuntu
version: "20.04"
eventLogger: file
hostname: xxxxxxxxxxxxx
idMappings:
gidmap: null
uidmap: null
kernel: 4.19.128-microsoft-standard
linkmode: dynamic
memFree: 21707079680
memTotal: 26783453184
ociRuntime:
name: crun
package: 'crun: /usr/bin/crun'
path: /usr/bin/crun
version: |-
crun version 0.19.1.3-9b83-dirty
commit: 33851ada2cc9bf3945915565bf3c2df97facb92c
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
selinuxEnabled: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 7516192768
swapTotal: 7516192768
uptime: 54h 16m 19.97s (Approximately 2.25 days)
registries:
search:

• docker.io
• quay.io
  store:
  configFile: /etc/containers/storage.conf
  containerStore:
  number: 0
  paused: 0
  running: 0
  stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Using metacopy: "false"
  imageStore:
  number: 4
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 3.1.2
  Built: 0
  BuiltTime: Thu Jan 1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.1.2

[giuseppe]

Seems like that is correct

❯ ll /tmp | grep run
drwx------  7 lucas lucas  4.0K Apr 26 10:09 podman-run-1000
drwx------  3 lucas lucas  4.0K Mar  9 10:57 run-1000

thanks! Could you remove these directories and try again?

[istyf]

That solved it for me!

[rhatdan]

Ok then I will close, reopen if you think the upstream needs more progress.

[MLNW]

Thanks @giuseppe! That was it!