podman run --rm -it --userns=keep-id --privileged=true --arch arm64 fedora sudo fails while it works without --arch

[eyebeem-mikel]

Issue Description

Description:

When podman starts a privileged container with keeping the userns sudo works perfect until the option --arch is used.
I tried arm64 and s390x where sudo errors out with Error while loading /usr/bin/sudo: Permission denied
I could not isolate the error further sofar, ideas welcome. It is an hard error so I can try things...

Steps to reproduce the issue

Steps to reproduce the issue

1. run podman run --rm -it --userns=keep-id --privileged=true --arch arm64 fedora sudo which fails for me with Error while loading /usr/bin/sudo: Permission denied
2. run podman run --rm -it --userns=keep-id --privileged=true fedora sudo which works as expected

Describe the results you received

Sudo fails with some permission error, since the image content is exactly the same the problem most likely is in the runtime.

Describe the results you expected

Describe the results you expected

podman info output

### Versions:

podman version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.19.7
Built:        Fri Apr 14 17:42:56 2023
OS/Arch:      linux/amd64

podman version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.19.7
Built:        Fri Apr 14 17:42:56 2023
OS/Arch:      linux/amd64
mandres:zaci_carpo$ ^C
mandres:zaci_carpo$ podman info
host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.59
    systemPercent: 0.82
    userPercent: 4.59
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  hostname: traktor
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.2.11-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7701835776
  memTotal: 33327575040
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.3-2.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 8h 23m 49.00s (Approximately 0.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/mandres/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/mandres/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 122898747392
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/mandres/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486976
  BuiltTime: Fri Apr 14 17:42:56 2023
  GitCommit: ""
  GoVersion: go1.19.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

podman-4.5.0-1.fc37.x86_64

### Podman in a container

No

### Privileged Or Rootless

Privileged

### Upstream Latest Release

Yes

### Additional environment details

N/A

### Additional information

N/A

[eyebeem-mikel]

The problem can be further reduced by using:
OK: podman run --rm -it --userns=keep-id fedora sudo
KO: podman run --rm -it --userns=keep-id --arch arm64 fedora sudo
Attention, when switching the platform you need to delete the fedora image first in your local registry otherwise you will get a platform mismatch.

[Luap99]

Does it work without the --userns argument?

[eyebeem-mikel]

Yes, it does.
OK: podman run --rm -it --arch s390x fedora sudo
OK: podman run --rm -it --userns=keep-id fedora sudo
KO: podman run --rm -it --userns=keep-id --arch arm64 fedora sudo

[Luap99]

@nalind @giuseppe PTAL

[nalind]

Sounds like an expected limitation when qemu-user-static is registered with binfmt_misc with the F flag, but not the C flag, per the kernel doc. I guess we'll have to remember that one.

[eyebeem-mikel]

@nalind thanks for looking into this. Does that mean this works as designed?
From a user perspective it would be kind of wired since the expectation is that a emulation works like on the real hardware just slower. Is this something which can be fixed in the future?
Currently for my specific case it looks like I can get around that issue by not using userns. Anyways, in the future it would be great if that can be used again since it makes it so much easier to work with mapped folders.
Thanks for your efforts.

[nalind]

@nalind thanks for looking into this. Does that mean this works as designed? From a user perspective it would be kind of wired since the expectation is that a emulation works like on the real hardware just slower. Is this something which can be fixed in the future?

This specific case is a configuration setting, and my assumption is that the packaging-level decision to not also set the "C" flag when registering the interpreter was made with care, but on your own system, you're able to override that by editing the files under /usr/lib/binfmt.d.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

Closing as this doesn't seems to be a podman issue.