Unexpected different behavior between rootless and root podman

[andrescodas]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. this works: podman run --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon" > log.txt
2. this fails: sudo podman run --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon" > sudo_log.txt

Describe the results you received:

chfn: PAM: System error
adduser: `/usr/bin/chfn -f Avahi mDNS daemon avahi' returned error code 1. Exiting.
dpkg: error processing package avahi-daemon (--configure):
 installed avahi-daemon package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of libnss-mdns:amd64:
 libnss-mdns:amd64 depends on avahi-daemon (>= 0.6.16-1); however:
  Package avahi-daemon is not configured yet.

dpkg: error processing package libnss-mdns:amd64 (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Processing triggers for dbus (1.12.2-1ubuntu1.2) ...
Errors were encountered while processing:
 avahi-daemon
 libnss-mdns:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

see attached the full logs
sudo_log.txt
log.txt

Describe the results you expected:

both steps above 1 and 2 with same behavior, i.e., successful installation of avahi-deamon

Additional information you deem important (e.g. issue happens only occasionally):

This worked fine 1 month ago...

Output of podman version:

podman version 3.0.2-dev

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.8
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-1.module+el8.4.0+10607+f4da7515.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: b883692702312720058141f16b6002ab26ead2e7'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: caperucita
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-305.el8.x86_64
  linkmode: dynamic
  memFree: 17397620736
  memTotal: 33448177664
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rc92.module+el8.4.0+10607+f4da7515.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+10607+f4da7515.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 33227272192
  swapTotal: 33227272192
  uptime: 2h 8m 26.45s (Approximately 0.08 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/codas/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-2.module+el8.4.0+10607+f4da7515.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/codas/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/codas/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1617784614
  BuiltTime: Wed Apr  7 05:36:54 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.2-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

I'm going to assume the failure has something to do with the lack of a user namespace for the root container, given the failure looks like a PAM/adduser one. Can you try adding --userns=auto to the root command, to see if that resolves things? I also note that the version of Podman you're using is somewhat old - can you test with a newer version?

[andrescodas]

I get this error with your suggestion:

sudo podman run --userns=auto --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon"
ERRO[0000] cannot find mappings for user "containers": No subuid ranges found for user "containers" in /etc/subuid 
Error: error creating container storage: could not find enough available IDs

Now I'm running a dnf update on my rhel 8.4, it will upgrade podman to:

 Package                                            Architecture                  Version                                                            Repository                                               Size

 podman                                             x86_64                        3.0.1-7.module+el8.4.0+11311+9da8acfb                              rhel-8-for-x86_64-appstream-rpms                         12 M

[mheon]

Ah, that would be the most recent for RHEL.

@giuseppe What are the steps for enabling --userns=auto, again?

[rhatdan]

The distribution is supposed to allocate a large range of UIDs within /etc/subuid for containers.

grep containers /etc/subuid
containers:2147483647:2147483648

Auto uses this for finding ranges that it can use.

We should document this in podman run/create man pages.

[andrescodas]

Hi again...

I'm not sure if this issue was correctly closed by @openshift-merge-robot

I'm confused, and I'm not an expert on the uid mappings. grep containers /etc/subuid returns nothing here and there is also no user containers (grep containers /etc/passwd returns nothing)

Is this an error specific of my distro (rhel 8.4) and I have to report it somewhere else? Did I miss to configure something? How should I proceed?

[rhatdan]

Yes we just documented that you need to add a containers entry to /etc/subuid and /etc/subgid. I don't believe that podman should do this by default at install time.

[andrescodas]

the added documentation addresses this side issue related to the use of --userns=auto

the cause of the original issue:

this works: podman run --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon" > log.txt
this fails: sudo podman run --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon" > sudo_log.txt

is related to the lack of CAP_AUDIT_WRITE capabiltiies as described here. Then, this works:

sudo podman run --cap-add audit_write --rm -t ubuntu:bionic bash -c "apt update && apt install -y avahi-daemon"

this same question raised before remains

[rhatdan]

I am guessing (Hopefully somewhat educated) is that there is some check that understands a process is not root when run in a rootless container, whereas when it runs as real root, it fails.