failed to chown recursively host path: lchown: operation not permitted

[Qix-]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

After bringing my Mac back up from a sleep, all of a sudden I cannot mount a host directory into containers anymore:

Error: error preparing container <snip> for attach: failed to chown recursively host path: lchown /host: operation not permitted

Steps to reproduce the issue:

Not sure if this is reproducible perfectly but this is, from my view, how it happened:

1. Create a machine using podman machine init -v $HOME/podman:/host --now

2. Use the directory for e.g. MongoDB: mkdir -p $HOME/podman/mongo && podman run -ti --rm -v /host/mongo:/data:rw,U -u mongodb mongo --dbpath /data

Describe the results you received:

Error: error preparing container <snip> for attach: failed to chown recursively host path: lchown /host/mongo: operation not permitted

This does not occur when the U option is omitted, however the directory is no longer mounted with the correct user permissions and thus the directory cannot be written to.

Describe the results you expected:

This was working just a few hours ago, then I put my mac to sleep and now I have no idea what's changed. Mongo was properly working, I could exec in and read/write to the mount point, and it was correctly mounted as the -u user (mongodb).

Additional information you deem important (e.g. issue happens only occasionally):

N/A

Output of podman version:

Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Tue Jun 14 22:12:46 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.4
Built:        Fri Jul 22 21:06:49 2022
OS/Arch:      linux/arm64

Output of podman info:

host:
  arch: arm64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 96.48
    systemPercent: 2.18
    userPercent: 1.34
  cpus: 3
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.18.16-200.fc36.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 2632523776
  memTotal: 4090335232
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.aarch64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 5m 7.65s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106825756672
  graphRootUsed: 2931798016
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1658516809
  BuiltTime: Fri Jul 22 21:06:49 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman: stable 4.2.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.1.1 (174 files, 46.4MB) *
  Poured from bottle on 2022-08-10 at 15:10:09
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go-md2man ✘, [email protected] ✘
Required: qemu ✔
==> Options
--HEAD
        Install HEAD version
==> Caveats
fish completions have been installed to:
  /opt/homebrew/share/fish/vendor_completions.d
==> Analytics
install: 15,693 (30 days), 53,053 (90 days), 182,623 (365 days)
install-on-request: 15,563 (30 days), 52,600 (90 days), 182,138 (365 days)
build-error: 47 (30 days)

NOTE: Not sure if makes any difference but brew is reporting 4.2.0 but podman --version says 4.1.1 - is this deliberate?

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on MacOS 12.5 (21G72).

I've tried to re-init the machine (via podman machine stop and the above podman machine init ...) several times, restarted my machine, etc. Hasn't helped.

Also worth mentioning, this is pretty much a branch new M1 macbook I received for work literally yesterday. Likelihood of this being some non-standard environment thing is exceedingly low.

[rhatdan]

I am not sure we can fix this. The issue is Podman is attempting to chown the UID of the file on the MAC to something that is not the USERS UID and the MAC is saying you are not allowed. If you use a directory in the VM but not on the host, it should work fine. Or if you run the application in the container with the users UID, it should also work.

[Qix-]

The thing is... this worked like an hour before I wrote this issue. I have no idea what actually changed. It was working fine until I slept/woke the (physical) machine, then it all broke. The commands were being run from a shell script, so I there isn't the possibility of anything having changed between that time other than the machine having rebooted or slept/woke.

The directory I'm specifying with -v is indeed on the VM, not on the mac. I've initialized my container to expose a mac folder into the underlying machine VM using podman machine init -v /mac-dir:/host-dir, which works fine, then I do podman run -v /host-dir:/container-dir. There shouldn't be any problems there, in theory.

The problem is that, in my case, mongo does a setuid to another user, which is not the same user as the bind mount within the container itself, which means it's able to read the contents of the mountpoint but gets an EPERM whenever it tries to open files for writing. There doesn't appear to be a way to specify a file mode (e.g. 0777) to work around this, either.

Any ideas on how to debug this would be great. Like I said, it was working. I would run mongo, see the database and log files populate on my mac, and everything Just Worked. Otherwise, this is a huge blocker to move to podman as it means containers aren't able to proxy to the host mac machine for development purposes.

[rhatdan]

If you podman machine ssh into the machine and then execute the podman run command, does it work?

[Qix-]

No. Same error.

Error: failed to chown recursively host path: lchown /host/mongo: operation not permitted

[rhatdan]

What command inside of the VM are you running?
$ podman run -ti --rm -v /host/mongo:/data:rw,U,Z -u mongodb mongo --dbpath /data
If so could you try
$ mkdir /tmp/mongo
$ podman run -ti --rm -v /tmp/mongo:/data:rw,U,Z -u mongodb mongo --dbpath /data

[Qix-]

That of course works fine. It's when the host directory is mounted in via QEMU.

[rhatdan]

Right, which goes back to my original comment that You can not chown the mapping of the file on the home directory.
What we really need for this is a way to specify that the container USER should run as the users UID running Podman.

@giuseppe this is a use case for #15294

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since syncid got merged, I am closing this issue.