using --userns keep-id prevents container from starting up

[tmstn]

Issue Description

With some images, podman fails to start if --userns keepid is set. There is no error message and podman simply hangs until it is manually terminated.

Steps to reproduce the issue

Steps to reproduce the issue

1. Run podman run --log-level debug --rm --user 1001:1001 --userns=keep-id ubuntu:mantic echo "STARTED" where 1001 is the id of your host user.
2. podman hangs on overlay: mount_data=lowerdir=...

Describe the results you received

The container does not start but hangs

Describe the results you expected

The container should start. If I change the command to podman run --log-level debug --rm --user 1001:1001 --userns=keep-id alpine:latest echo "STARTED" it does start which leads me to believe that it is image specific, yet there is no debug information or errors to help fix this on my end.

podman info output

$ podman version
Client:       Podman Engine
Version:      5.1.2
API Version:  5.1.2
Go Version:   go1.22.5
Git Commit:   94a24974ab345324db1a1489c924af4b89d2d0e9
Built:        Thu Jul 11 22:26:12 2024
OS/Arch:      linux/amd64

$ podman info
host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 89.72
    systemPercent: 2.53
    userPercent: 7.75
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  freeLocks: 2040
  hostname: lenovox250
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.44-1-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 465281024
  memTotal: 8201465856
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.11.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.11.0
    package: /usr/lib/podman/netavark is owned by netavark 1.11.0-2
    path: /usr/lib/podman/netavark
    version: netavark 1.11.0
  ociRuntime:
    name: runc
    package: /usr/bin/runc is owned by runc 1.1.13-1
    path: /usr/bin/runc
    version: |-
      runc version 1.1.13
      spec: 1.0.2-dev
      go: go1.22.4
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_07_26.57a21d2-1
    version: |
      pasta 2024_07_26.57a21d2
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 7205949440
  swapTotal: 8589930496
  uptime: 21h 32m 20.00s (Approximately 0.88 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/tom/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tom/.local/share/containers/storage
  graphRootAllocated: 490792517632
  graphRootUsed: 212840861696
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 393
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/tom/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720733172
  BuiltTime: Thu Jul 11 22:26:12 2024
  GitCommit: 94a24974ab345324db1a1489c924af4b89d2d0e9
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[tmstn]

It seems that I was just being very impatient. The first run seems to take time with some images and appears to hang on overlay: mount_data=lowerdir=...

Some images take much longer than others, but subsequent runs are much faster.

I'm very sorry to waste your time.

[tmstn]

Scratch that. It is still happening with some images and the hang is way too long. It seems like a permissions issue but I obviously do not know what is causing it.

When I run podman run --log-level debug --rm --user 1001:1001 --userns keep-id golang:1 echo "STARTED" the same hanging occurs but this time what seems to be indefinitely (the image can be pulled with podman pull docker.io/golang:1).

The full log entry for the debug log is:
DEBU[0000] overlay: mount_data=lowerdir=/home/tom/.local/share/containers/storage/overlay/l/HGQXNXEUKH6WKGKR2MTLLD6ZPF:/home/tom/.local/share/containers/storage/overlay/l/D2QCTANCFRLBWPQ6RHKFKUFQET:/home/tom/.local/share/containers/storage/overlay/l/2AUFHXJUVY4HIVQNV7IYJNNWKD:/home/tom/.local/share/containers/storage/overlay/l/LXOHAVN62KUNE5BN63LNQA4ZHD:/home/tom/.local/share/containers/storage/overlay/l/SXMTGKGZ7AN7P5JOAWJX6IHYQT:/home/tom/.local/share/containers/storage/overlay/l/RIV5H6J46ZODGC4ZR5EMYNTWXX:/home/tom/.local/share/containers/storage/overlay/l/GDSTW3PQ6DVW3NWNP2O2SJ6KXS,upperdir=/home/tom/.local/share/containers/storage/overlay/6a40ff3dc8d4d5b6c2364371cb00054a3bd205f2ab8d58b69db7ef27784beddd/diff,workdir=/home/tom/.local/share/containers/storage/overlay/6a40ff3dc8d4d5b6c2364371cb00054a3bd205f2ab8d58b69db7ef27784beddd/work,,userxattr

In here I see that upperdir and workdir are set to sub directories in /home/tom/.local/share/containers/storage/overlay/6a40ff3dc8d4d5b6c2364371cb00054a3bd205f2ab8d58b69db7ef27784beddd/ but this subdirectory has been created with the user:group 100000:100000 which does not exist on my system and gives my user 1001 a permission denied error if I try to access it from the terminal.

When I manually terminated the process, this directory was removed from /home/tom/.local/share/containers/storage/overlay/.

I decided to run podman system reset to see if this would help at all but the same problem occurs albeit with a different random directory with bad user:group ownership set.

[tmstn]

Scratch that again, as the same process happens with ubuntu:mantic using the same user:group id (100000) and after waiting for some time, I can access those folders without a permission denied error.

I guess the best question to ask is why is there such a long delay when using --userns keep-id as opposed to not using it.

It's worth noting that there is a difference in the debug logs between images that worked eventually after a delay and those that seemingly hanged forever. These entries were just above the overlay: mount_data=lowerdir=... debug log entry:
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support

[rhatdan]

There is no idmap support for rootless, so when you pull with one user namespace and then switch to a different namespace with --userns=keep-id, then container storage needs to chown -R every file in the image. This can take a period of time. Once this is done once, all further use of the image will not need to chown again. In rootful containers we can use idmap, which makes the chowning instantanious.

[tmstn]

Oh okay, thank you. Most of the time I run without --userns keep-id set so my host user is mapped to root on the container but there are some edge cases where it is unavoidable. I will pull those images and leave it running overnight.

[eriksjolund]

In the output of the command

podman version

that is shown in #23651 (comment) there are the lines

graphDriverName: overlay

and

 Native Overlay Diff: "true"

This tells us that native overlayfs is used.

You could also consider using fuse-overlayfs to see if it speeds up the creation of the container.
One idea could be to create another user account and perform some benchmarks.

The Podman performance guide mentions that creating a container is faster
with fuse-overlayfs than with native overlayfs if these conditions are met

• rootless Podman is used
• a modified UID/GID mapping is used
• no container has yet been created with the specified container image and UID/GID mapping

In the text there is also a link to a #16541 (comment) which contains some more information.