Can't pull read only image to additionalimagestores as root

[StarpTech]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Hi, I'm running podman inside podman. I run my container in privileged mode:

podman run -v "./.cache:/var/lib" --privileged <image>

Inside the container I use podman 4.2.0. I want to pull an image to the read-only store but I run into the following error:

podman --storage-opt --root /var/lib/shared pull <image>

Output:

15:48:27.980  [pull-base-images]: Getting image source signatures
15:48:27.980  [pull-base-images]: Copying blob sha256:6f39401b45edd5dea4ae0e4a11d9ee330c2cc1211c816231f37a962583575b71
15:48:27.980  [pull-base-images]: Copying blob sha256:45d69f7b887ee79101f69183932914d65c0b6c6f0f3accae9910f57a0db09b39
15:48:27.980  [pull-base-images]: Copying blob sha256:3e6080001d7b2e588ba7bd7c83b4fe5cdc389c5619525db7f24656198f7d44ab
15:48:27.980  [pull-base-images]: Copying blob sha256:6286be2976b295235eb550af1a40c60dbbdba0d0603821020d41b282014fbafb
15:48:27.980  [pull-base-images]: Copying blob sha256:2bd1b6afc5aedfcdd107a8b861836e27b9c0f8e15bc79bd8b3dc1bb2470f59a3
15:48:27.980  [pull-base-images]: Copying blob sha256:0e2f374cd7e26ca944b6b2d064b7730b09ed50a75102307287ad8bc7e00f7f54
15:48:27.982  [pull-base-images]: Copying blob sha256:163ab8cb719554d9257f6ca63d861465498d4d97b273a435de871e1613321ab2
15:48:30.001  [pull-base-images]: Error: writing blob: adding layer with blob "sha256:45d69f7b887ee79101f69183932914d65c0b6c6f0f3accae9910f57a0db09b39": processing tar file(operation not permitted): exit status 1

Running just podman pull <image> works.

Steps to reproduce the issue:

1. Follow https://www.redhat.com/sysadmin/podman-inside-container and https://www.redhat.com/sysadmin/image-stores-podman

2. Try to pull an image to the read-only store as root.

Describe the results you received:

I can't store the pulled image in the store

Describe the results you expected:

I'd expect in privileged mode, that there should be no permission error.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

3.4.4

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 32
  distribution:
    codename: jammy
    distribution: pop
    version: "22.04"
  eventLogger: journald
  hostname: pop-os
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.19.0-76051900-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 22502309888
  memTotal: 67347271680
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 4294434816
  swapTotal: 4294434816
  uptime: 8h 20m 52.6s (Approximately 0.33 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/starptech/.config/containers/storage.conf
  containerStore:
    number: 48
    paused: 0
    running: 0
    stopped: 48
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/starptech/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 655
  runRoot: /run/user/1000/containers
  volumePath: /home/starptech/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[StarpTech]

I could find the fix. You have to pass the mount_program option to storage option --storage-opt", "overlay.mount_program=/usr/bin/fuse-overlayfs. Make sense.

[giuseppe]

opened a PR: #1312

[StarpTech]

@giuseppe could you give me a brief summary of why this PR fixed the issue?