podman cannot be run in amd64 container running under OSX arm processor with rosetta

[marvingreenberg]

Issue Description

I've created a UBI9 based container to use in certain contexts, with podman installed (dnf install -y podman). I am on an Arm mac as everyone will be someday, running containers using Docker Desktop. I'm building x86-64 containers since that's the target platform, so I run them under Rosetta on osx.

The apparent way that (Docker Desktop) integration happens is somehow using rosetta to run every command. Apparently podman is looking at some absolute count to get the command argument and is failing. The scenario below shows this

$ uname -a
Darwin marvins-laptop.lan 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64

$ docker run -u root -e --platform  linux/amd64 -it builder-jenkins:ubi9.2.v3

builder_user ~ $ rpm -q podman
podman-4.6.1-7.el9_3.x86_64
builder_user ~ $ pwd
/home/builder_user
builder_user ~ $ podman info
Error: unrecognized command `podman podman`
Try 'podman --help' for more information
builder_user ~ $ podman --version
podman version 4.6.1
builder_user ~ $ podman info& podman info& ps -ef& podman info&
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 11:20 pts/0    00:00:00 /rosetta/rosetta /usr/bin/su su builder_user -- -c bash
builder+    64     1  0 11:20 pts/0    00:00:00 /rosetta/rosetta /usr/bin/bash bash
builder+   110     1  0 11:20 ?        00:00:00 /rosetta/rosetta /usr/libexec/podman/catatonit catatonit -P
builder+   153    64  0 11:21 pts/0    00:00:00 /rosetta/rosetta /usr/bin/podman podman info
builder+   154    64  0 11:21 pts/0    00:00:00 /rosetta/rosetta /usr/bin/podman podman info
builder+   155    64  0 11:21 pts/0    00:00:00 /usr/bin/ps ps -ef

Steps to reproduce the issue

Steps to reproduce the issue

1. Build a UBI9 container target, for target arch linux/amd64, with podman installed
2. On an ARM OSX system with rosetta installed and integrated into docker desktop, run the container
3. run podman info

A simple dockerfile is below.

FROM --platform=linux/amd64 registry.access.redhat.com/ubi9/ubi:9.2
RUN dnf update -y && dnf install -y podman procps-ng
RUN useradd -U builder_user --shell /bin/bash
USER builder_user
WORKDIR /home/builder_user/
CMD [ "bash","-l" ]

Describe the results you received

$ podman info
Error: unrecognized command podman podman

Describe the results you expected

I expect it to run. I understand that podman in this environment may not work properly - for example starting podman machine may fail, other things may not function. Interestingly this behavior is NOT observed for podman machine info, etc. For example, podman machine init correctly parses the command line, and reports that it cannot start a VM.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

UBI9 container, linux amd64 arch 0n OSX Arm64

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Looks like your setup is broken, podman podman is not a valid command so we error correctly.
Your ps output even shows /usr/bin/su su ... and /usr/bin/bash bash so this is not related to podman at all, it is just a coincidence that bash and su works as they just execute themselves again.

[afbjorklund]

I think that is normal, how programs look when run through a binary translator

[afbjorklund]

It looks the same way, when running with qemu-user (instead of rosetta)

$ qemu-aarch64 bin/podman-remote-static-linux_arm64 podman-remote-static info
Error: unrecognized command `podman-remote-static-linux_arm64 podman-remote-static`
Try 'podman-remote-static-linux_arm64 --help' for more information

So there is something strange, in how "info" is trying to invoke another podman

I'm not sure it is a big thing, since it is something that you would not normally do.

---

But does it work OK on Linux, if you use binfmt_misc instead of invoking it yourself.

bin/podman-remote-static-linux_arm64 info

[Luap99]

Can you add --log-level debug, it should give us the cli args we see.

[afbjorklund]

Seems like it would be easier to run Podman Desktop, instead of podman in DD?

i.e. to use an linux/arm64 OS, and then build linux/amd64 with Podman (and qemu)

[paulcalabro]

I'm seeing this as well when running Podman within a Alpine image:

INFO[0000] /usr/bin/podman filtering at log level debug 
DEBU[0000] Called podman.PersistentPreRunE(/usr/bin/podman podman --log-level debug load) 
Error: unrecognized command `podman podman`
Try 'podman --help' for more information

[rhatdan]

This is most likely a rosetta issue, not something Podman can fix. Emulation tools do not seem to like tools reexecing themselves.

[tnk4on]

@marvingreenberg This has been fixed in Podman v5.1.0.

% podman -v
podman version 5.1.0
% podman machine inspect --format {{.Rosetta}}
true
% cat > Containerfile <<EOF
FROM --platform=linux/amd64 registry.access.redhat.com/ubi9/ubi:9.2
RUN dnf update -y && dnf install -y podman procps-ng
RUN useradd -U builder_user --shell /bin/bash
USER builder_user
WORKDIR /home/builder_user/
CMD [ "bash","-l" ]
EOF
% podman build --pull -t emu-test .
% podman run --rm -it --arch amd64 emu-test podman -v
podman version 4.9.4-rhel