podman build via stdin fails in remote rootless

[sstosh]

Issue Description

Image build with podman-remote build -f - via stdin in remote rootless is failed
due to permission denied.
Maybe related to #17480.

Steps to reproduce the issue

1. Create a libpod_lock, e.g. # podman ps

$ ls -l /dev/shm/
total 168
-rw-------. 1 root root 82488 Feb 13 10:53 libpod_lock
-rw-------. 1 test test 82488 Feb 13 17:28 libpod_rootless_lock_1000

2. Image build with podman-remote build -f - via stdin

$ echo "from alpine" | podman-remote build -f -
ERRO[0000] 1 error occurred:
        * open /dev/shm/libpod_lock: permission denied


Error: Post "http://d/v4.5.0/libpod/build?dockerfile=%5B%22%2Fvar%2Ftmp%2Fbuild4259015452%22%5D&forcerm=1&httpproxy=1&identitylabel=1&idmappingoptions=%7B%22HostUIDMapping%22%3Atrue%2C%22HostGIDMapping%22%3Atrue%2C%22UIDMap%22%3A%5B%5D%2C%22GIDMap%22%3A%5B%5D%2C%22AutoUserNs%22%3Afalse%2C%22AutoUserNsOpts%22%3A%7B%22Size%22%3A0%2C%22InitialSize%22%3A0%2C%22PasswdFile%22%3A%22%22%2C%22GroupFile%22%3A%22%22%2C%22AdditionalUIDMappings%22%3Anull%2C%22AdditionalGIDMappings%22%3Anull%7D%7D&isolation=3&jobs=1&layers=1&networkmode=0&nsoptions=%5B%7B%22Name%22%3A%22user%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%5D&omithistory=0&outputformat=application%2Fvnd.oci.image.manifest.v1%2Bjson&platform=%2F&pullpolicy=missing&rm=1&seccomp=%2Fusr%2Fshare%2Fcontainers%2Fseccomp.json&shmsize=67108864&t=": io: read/write on closed pipe

Describe the results you received

See above.

Describe the results you expected

Image build via stdin succeeds.

$ echo "from alpine" | podman-remote build -f -
STEP 1/1: FROM alpine
COMMIT
--> b2aa39c304c
b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0-dev
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 99.87
    systemPercent: 0.05
    userPercent: 0.09
  cpus: 12
  distribution:
    distribution: fedora
    variant: server
    version: "37"
  eventLogger: journald
  hostname: fedora37
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.1.9-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2670391296
  memTotal: 8307339264
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-3.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8202743808
  swapTotal: 8306814976
  uptime: 120h 16m 49.00s (Approximately 5.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  localhost:35895:
    Blocked: false
    Insecure: true
    Location: localhost:35895
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:35895
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/test/.local/share/containers/storage
  graphRootAllocated: 101938364416
  graphRootUsed: 21018914816
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0-dev
  Built: 1676253199
  BuiltTime: Mon Feb 13 10:53:19 2023
  GitCommit: f099c1fc9a840067ac0c98c1770a45fd378a07d8
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0-dev

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

KVM

Additional information

• podman-remote build - succeeds the image build

$ echo "from alpine" | podman-remote build -
STEP 1/1: FROM alpine
COMMIT
--> b2aa39c304c
b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d

• podman-remote build -f - succeeds if remove libpod_lock.

$ sudo rm /dev/shm/libpod_lock

$ ls -l /dev/shm/
total 84
-rw-------. 1 test test 82488 Feb 14 10:03 libpod_rootless_lock_1000

$ echo "from alpine" | podman-remote build -f -
STEP 1/1: FROM alpine
COMMIT
--> b2aa39c304c
b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d

[jordansissel]

@sstosh I'm able to almost able to reproduce your report. I have isolated my problem to podman machine or podman's --url feature to talk to a podman machine. I suspect my problem (#17480) and yours are related, though I am unfamiliar with podman's internals, so this is just a guess at this time.

To reproduce: On Linux with podman 4.3.1, this works fine for me:

echo "from scratch" | podman build -f -

However, if I create a podman machine and talk to it instead, it fails:

% podman machine init
% podman machine start
....
API forwarding listening on: /home/jls/.local/share/containers/podman/machine/podman-machine-default/podman.sock
...

# This fails
% echo "from scratch" | podman --url 'unix:///home/jls/.local/share/containers/podman/machine/podman-machine-default/podman.sock' build -f -
ERRO[0000] 1 error occurred:
	* open /dev/shm/lttng-ust-wait-8-42: permission denied

[jordansissel]

I noticed a possibly related issue on my mac when I have visual studio code running (addmittedly in an oddly installed way). My mac would mount the 'Visual Studio Code.app' as a filesystem mount, then for some reason podman build would try to read files from this directory for /no apparent reason/ and fail with this error:

readdir: fts_read: Result too large

My next hypothesis is that podman build is enumerating all mount points and attempting to read the files in every mount point.

I think when no context is provided, podman build has a little freak out (this is not a technical term) and attempts to read a bunch of completely unrelated files as context.

I test this hypothesis by comparing a build step with and without context, when giving -f -

Without context:

% echo "from scratch" | strace -e trace=file -fo /tmp/x podman --url 'unix:///home/jls/.local/share/containers/podman/machine/podman-machine-default/podman.sock' build -f -
% echo "from scratch" | strace -e trace=file -fo /tmp/x podman --url 'unix:///home/jls/.local/share/containers/podman/machine/podman-machine-default/podman.sock' build -f -  
ERRO[0000] 1 error occurred:
	* open /dev/shm/lttng-ust-wait-8-42: permission denied

With context of "." (current directory):

% echo "from scratch" | strace -e trace=file -fo /tmp/x podman --url 'unix:///home/jls/.local/share/containers/podman/machine/podman-machine-default/podman.sock' build -f - .
STEP 1/1: FROM scratch
COMMIT
--> 611069e3699
611069e36993edc066ace0678be87c85fb179614873ea3c6fcc878ec80381a0c

When context is set to . it seems to work.

So what does the strace look like? Well... it's quite busy:

% wc -l /tmp/podman-*
  1093 /tmp/podman-no-context
   274 /tmp/podman-with-context

Without context, podman does ~800 more file-related syscalls. Looking at just openat, it seems to be walking most (all?) of the filesystem (starting at /dev, etc) and eventually failing when it tries to read something that is rejected by access controls.