Mounted volumes owned by root rather than container user

[evankanderson]

Issue Description

On MacOS (at least), volume mounts appear to be owned by root when running containers (such as neo4j) which run as a non-root user and attempt to create/chown files.

With podman:

$ podman run -v $HOME/neo4j/data:/data --entrypoint /bin/bash neo4j:latest -c "ls -la /data; whoami"

total 0
drwxr-xr-x. 2 root nogroup 64 Sep 17 08:46 .
dr-xr-xr-x. 1 root root    28 Sep 17 08:46 ..
root

$ podman run -v $HOME/neo4j/data:/data neo4j:latest                          
Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
chown: changing ownership of '/data': Operation not permitted

With docker:

$ docker run -v $HOME/neo4j/data:/data --entrypoint /bin/bash neo4j:latest -c "ls -la /data; whoami"

total 4
drwxr-xr-x 2 root root   64 Sep 17 08:46 .
drwxr-xr-x 1 root root 4096 Sep 17 08:46 ..
root

$ docker run -v $HOME/neo4j/data:/data neo4j:latest                                           
Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
2023-09-17 08:47:56.209+0000 INFO  Logging config in use: File '/var/lib/neo4j/conf/user-logs.xml'
2023-09-17 08:47:56.218+0000 INFO  Starting...
2023-09-17 08:47:56.792+0000 INFO  This instance is ServerId{eddd2281} (eddd2281-d833-473a-aaea-ab69a8119230)

Two differences I notice:

1. The podman directory is owned by nogroup rather than by root.
2. The podman directory listing has a different total number of entries (0 vs 4)

Steps to reproduce the issue

Steps to reproduce the issue

(See the issue description)

2. podman run -v $HOME/neo4j/data:/data neo4j:latest
3. The container fails to run on podman, but succeeds on Docker Desktop

Describe the results you received

Running under podman:

Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
chown: changing ownership of '/data': Operation not permitted

Describe the results you expected

Running under docker:

Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
2023-09-17 08:47:56.209+0000 INFO  Logging config in use: File '/var/lib/neo4j/conf/user-logs.xml'
2023-09-17 08:47:56.218+0000 INFO  Starting...
2023-09-17 08:47:56.792+0000 INFO  This instance is ServerId{eddd2281} (eddd2281-d833-473a-aaea-ab69a8119230)

podman info output

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.81
    systemPercent: 0.11
    userPercent: 0.08
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2040
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 5907152896
  memTotal: 6762696704
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.7-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230823.ga7e4bfb-1.fc38.aarch64
    version: |
      pasta 0^20230823.ga7e4bfb-1.fc38.aarch64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 22h 14m 45.00s (Approximately 0.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 2
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 99252940800
  graphRootUsed: 7177838592
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1693251554
  BuiltTime: Mon Aug 28 12:39:14 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Running on MacOS

### Additional information

_No response_

[rhatdan]

Podman on MAC currently does not support chowning the container, because it is built on PLAN9 file system for volume support. Podman is moving to Apple Native virtualization, hopefully in the next release 4.8. Which uses virtiofsd for its volumes, and I believe this supports chowning of files.

You can futz around with the --userns flag on podman to set the default user of the container to match your host user, which should eliminate the chown.

   ‐‐userns=mode
       Set the user namespace mode for the container. It defaults to the PODMAN_USERNS environment variable.  An  empty  value  ("")
       means user namespaces are disabled unless an explicit mapping is set with the ‐‐uidmap and ‐‐gidmap options.

       This option is incompatible with ‐‐gidmap, ‐‐uidmap, ‐‐subuidname and ‐‐subgidname.

       Rootless user ‐‐userns=Key mappings:

       ┌─────────────────────────┬───────────┬───────────────────────────────────┐
       │ Key                     │ Host User │ Container User                    │
       ├─────────────────────────┼───────────┼───────────────────────────────────┤
       │ ""                      │ $UID      │ 0 (Default User account mapped to │
       │                         │           │ root user in container.)          │
       ├─────────────────────────┼───────────┼───────────────────────────────────┤
       │ keep‐id                 │ $UID      │ $UID  (Map  user  account to same │
       │                         │           │ UID within container.)            │
       ├─────────────────────────┼───────────┼───────────────────────────────────┤
       │ keep‐id:uid=200,gid=210 │ $UID      │ 200:210  (Map  user  account   to │
       │                         │           │ specified  UID,  GID value within │
       │                         │           │ container.)                       │
       ├─────────────────────────┼───────────┼───────────────────────────────────┤
       │ auto                    │ $UID      │ nil (Host User UID is not  mapped │
       │                         │           │ into container.)                  │
       ├─────────────────────────┼───────────┼───────────────────────────────────┤
       │ nomap                   │ $UID      │ nil  (Host User UID is not mapped │
       │                         │           │ into container.)                  │
       └─────────────────────────┴───────────┴───────────────────────────────────┘

[rhatdan]

@baude WDYT?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rijenkii]

This is not MacOS-only issue, I think I am able to reproduce on Fedora 40:

$ podman run -v $HOME/neo4j/data:/data --entrypoint /bin/bash neo4j:latest -c "ls -lad /data; whoami"
drwxr-xr-x. 1 root root 0 Dec  9 11:52 /data
root

$ docker run -v $HOME/neo4j/data:/data --entrypoint /bin/bash neo4j:latest -c "ls -lad /data; whoami"
drwxr-xr-x. 1 1000 1000 0 Dec  9 11:52 /data
root

podman info

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 83.71
    systemPercent: 6.04
    userPercent: 10.26
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "39"
  eventLogger: journald
  freeLocks: 2033
  hostname: rijenkiipc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.11-300.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1075380224
  memTotal: 32971603968
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.1-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1084f9527c143699b593b44c23555fb3cc4ff2f3
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc39.x86_64
    version: |
      pasta 0^20231004.gf851084-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8326230016
  swapTotal: 8589930496
  uptime: 176h 28m 3.00s (Approximately 7.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/rijenkii/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rijenkii/.local/share/containers/storage
  graphRootAllocated: 498403901440
  graphRootUsed: 257035194368
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 31
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/rijenkii/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 21:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

[rhatdan]

Docker is running a rootful container, while Podman is running a rootless container. In both cases the actual UID of the data directory is 1000, But in the podman case you are runing within a user namespace with UID 1000 mapped to UID 0, which is why it looks like the file is owned by root.

To see the same behaviour as Docker, you could run

$ sudo podman run -v $HOME/neo4j/data:/data --entrypoint /bin/bash neo4j:latest -c "ls -lad /data; whoami"

If you ran Docker in rootless mode, you would see the same thing.

[rhatdan]

To play with User namespace try

$ podman unshare

You will see all of the files owned by UID 1000 are now listed as owned by root.

Take a look at my book Podman in Action for full description of User Namespace.