Volumes are resolved on client side on build but vm side on run on macOS

[jedvardsson]

Issue Description

It seems that volumes (--volume) are resolved differently on build vs run when running macOS. build checks that the file exists on the client-side (macOS) while run checks that the file exists on the remote side (podman machine vm side).

Steps to reproduce the issue

Steps to reproduce the issue

$ podman run -it --security-opt label=disable -v /run/user/501/podman/podman.sock:/run/podman/podman.sock --rm podman podman --url unix:///run/podman/podman.sock ps
CONTAINER ID  IMAGE                         COMMAND               CREATED                 STATUS                     PORTS       NAMES
ae29c1e55cc7  quay.io/podman/stable:latest  podman --url unix...  Less than a second ago  Up Less than a second ago              great_joliot
$ podman build --no-cache --security-opt label=disable -v /run/user/501/podman/podman.sock:/run/podman/podman.sock - < <(printf 'FROM podman\nRUN podman --url unix:///run/podman/podman.sock ps\n')
Error: stat /run/user/501/podman/podman.sock: no such file or directory
$ 

Describe the results you received

Podman build fails to find the source volume file because it checks on the client side on not the vm side.

Describe the results you expected

The source file should not be required to exists on client-side but only on the vm side. Compare to executing the steps on the vm instead.

$ podman machine ssh
[core@localhost ~]$ podman run -it --security-opt label=disable -v /run/user/501/podman/podman.sock:/run/podman/podman.sock --rm podman podman --url unix:///run/podman/podman.sock ps
CONTAINER ID  IMAGE                         COMMAND               CREATED                 STATUS                     PORTS       NAMES
c0205b6e3c89  quay.io/podman/stable:latest  podman --url unix...  Less than a second ago  Up Less than a second ago              loving_villani
[core@localhost ~]$ podman build --no-cache --security-opt label=disable -v /run/user/501/podman/podman.sock:/run/podman/podman.sock - < <(printf 'FROM podman\nRUN podman --url unix:///run/podman/podman.sock ps\n')
STEP 1/2: FROM podman
STEP 2/2: RUN podman --url unix:///run/podman/podman.sock ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
COMMIT
--> effd60c022e
effd60c022ee19dcc3eb5d54f4935656c3cf109bd0477230d40a32c1dac59194
[core@localhost ~]$ exit

podman info output

host:
  arch: arm64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 99.77
    systemPercent: 0.13
    userPercent: 0.1
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.0.18-300.fc37.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 566206464
  memTotal: 2049794048
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-3.fc37.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 35h 11m 23.00s (Approximately 1.46 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 4899102720
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 36
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668178831
  BuiltTime: Fri Nov 11 16:00:31 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Homebrew install on MacOS Ventura 13.1.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mheon]

I think the run behavior is more correct here - machine should ensure that appropriate directories are mounted into the VM to ensure server-side resolution works. @baude @ashley-cui Do you concur?

[Luap99]

I think I changed this recently, this should be tested with podman 4.4.
see #16777

[rhatdan]

@flouthoc PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[flouthoc]

This is not fixed, I can still reproduce in upstream. Checking it now.

[flouthoc]

Following PR should close this: containers/buildah#4792