Podman can't run command on M1 with container built on X86

[cscetbon]

Docker can run on Apple M1 a container built on X86 but Podman can't

$ docker --version
Docker version 20.10.8, build 3967b7d

$ docker run -ti senseyeio/alpine-aws-cli echo ok
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
ok

$ podman --version
podman version 3.4.1

$ podman machine list 
NAME                     VM TYPE     CREATED         LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  qemu        34 minutes ago  Currently running  1           2.147GB     10.74GB

$ podman run -ti senseyeio/alpine-aws-cli echo ok
{"msg":"exec container process `/bin/echo`: Exec format error","level":"error","time":"2021-10-31T01:54:47.000542236Z"

[rhatdan]

@baude If we install qemu-user-static on the Fedora CoreOS, this would just work, although would be slow.

[cscetbon]

@baude If we install qemu-user-static on the Fedora CoreOS, this would just work, although would be slow.

@rhatdan, do you mean there is a way to be more efficient or that's just what it is ? I suppose we can also mimic what is done by docker ?

[afbjorklund]

Docker uses the same qemu-user-static, and the same binfmt_misc feature of Linux in order to do this.

$ docker run -it arm64v8/alpine
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested
standard_init_linux.go:228: exec user process caused: exec format error
$ sudo apt install qemu-user-static
...
$ docker run -it arm64v8/alpine
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested
/ # exit

It is statically linked, so it works the same inside the container as it does in Linux outside the container.

$ podman machine init
Downloading VM image: fedora-coreos-34.20211016.2.1-qemu.x86_64.qcow2.xz: done  
Extracting compressed file
$ podman machine start
INFO[0000] waiting for clients...                       
INFO[0000] listening tcp://0.0.0.0:7777                 
INFO[0000] new connection from @ to /run/user/1000/podman/qemu_podman-machine-default.sock 
Waiting for VM ...
Machine "podman-machine-default" started successfully
$ podman-remote run -it amd64/alpine
Resolving "amd64/alpine" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/amd64/alpine:latest...
Getting image source signatures
Copying blob sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e
Copying config sha256:14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab
Writing manifest to image destination
Storing signatures
/ # exit
$ podman-remote run -it arm64v8/alpine
Resolving "arm64v8/alpine" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/arm64v8/alpine:latest...
Getting image source signatures
Copying blob sha256:552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9
Copying blob sha256:552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9
Copying config sha256:bb3de5531c18f185667b0be0e400ab24aa40f4440093de82baf4072e14af3b84
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/bin/sh`: Exec format error","level":"error","time":"2021-10-31T17:29:30.000710590Z"}

podman machine ssh sudo rpm-ostree install qemu-user-static
podman machine ssh sudo systemctl reboot

$ podman-remote run -it arm64v8/alpine
/ # exit

[cscetbon]

Then I guess it's the way to go and easy enough to implement 🙌

[afbjorklund]

Then I guess it's the way to go and easy enough to implement raised_hands

Note that you get much better performance, by selecting the same arch...

The "multi-arch" images (manifests) makes this easy, so that it gets the same name.
I used the arch-specific name above, just to force a mismatch for the demonstration.

https://github.com/docker-library/official-images#architectures-other-than-amd64

[cscetbon]

Yeah yeah I get it. Sometimes images are not available on one architecture and it's easier to just use them instead of having to build one for the arm64 architecture, especially for non production images.

[afbjorklund]

Probably one could have saved some space in the image, if not all binaries were in the same package ?

Most users are only interested in qemu-aarch64-static and qemu-x86_64-static, not the other 32 arch.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This seems like more of a discussion then an actual issue, unless you want us to ship qemu-user-static inside of the fedora coreos?

[gbraad]

@rhatdan is this something we want to consider? There is a case for some images to do so, like the IBM Java images.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[jarrellmark]

I'd recommend having podman machine ship with qemu-user-static by default.

Shipping with qemu-user-static will make the transition to Apple M1 smoother. There could be a lot of people switching over to podman with the discontinuation of free Docker Desktop. Would be good to make that a nice experience.

I recently was upgraded to macOS M1 at work and had to do some digging around to get our production docker containers running locally for testing.

Agreed that multi-arch is the way to go, but there should be a fallback enabled by default.

[baude]

@jarrellmark unfortunately that is not entirely in our team's hands. we are hopeful however.

[gbraad]

it is something we might wanna record in a new issue?

[ssbarnea]

I think that the issue is clear. We must be sure that out machines install with qemu by default.

We need to include a build time test that validates that the machine can execute both arm64 and amd64 images, so we would not have regressions.

[rhatdan]

@dustymabe FYI

[dustymabe]

There's a process for requesting packages in FCOS. You can make a request here

[jarrellmark]

@rhatdan , would it be possible for podman's ignition script to do something like sudo rpm-ostree install qemu-user-static, instead of having qemu-user-static be installed by default into FCOS?

[jarrellmark]

Made a FCOS package request here: coreos/fedora-coreos-tracker#1088

[kjujik]

After installation of qemu-user-static and rebooting VM it lost all mounted directories. Do you know how to handle it?

[mattih]

Yep, this seems to be happening on reboot, making the fix unusable as the mounted volume seems to be required on Mac.

[afbjorklund]

You might have to use "stop" and "start", instead of issuing the reboot command. Volumes might move to a service, in the future.

[ssbarnea]

While installing qemu-user-static seems to be working in the default machine (v36), i wish this would have not be needed and be implicit.

$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
sudo rpm-ostree install qemu-user-static
...
Removed:
  moby-engine-20.10.14-1.fc36.aarch64
Added:
  ipxe-roms-qemu-20200823-8.git4bd064de.fc36.noarch
  mpdecimal-2.5.1-3.fc36.aarch64
  python-pip-wheel-21.3.1-2.fc36.noarch
  python-setuptools-wheel-59.6.0-2.fc36.noarch
  python-unversioned-command-3.10.4-1.fc36.noarch
  python3-3.10.4-1.fc36.aarch64
  python3-libs-3.10.4-1.fc36.aarch64
  qemu-common-2:6.2.0-9.fc36.aarch64
  qemu-user-static-2:6.2.0-9.fc36.aarch64
Changes queued for next boot. Run "systemctl reboot" to start a reboot

That approach has several downsides:

• user needs to remember to perform something like podman machine ssh bash -c "sudo rpm-ostree install qemu-user-static && sudo systemctl reboot"
• need to reboot the VM
• is removal of moby-engine ok?
• overall experience is not really... "it just works"

[baude]

@dustymabe this is an ongoing theme for us ... how close is the layer stuff?

[dustymabe]

Not close enough. I'd still suggest following up here: https://bugzilla.redhat.com/show_bug.cgi?id=2061584#c22

[ssbarnea]

To be honest, I do not think that 40MB is a real problem or a good enough reason for not just installing qemu-user-static. It would be far more useful to make it work ASAP and we can look into size optimization later. There are considerable maintenance costs if they do the split, not to mention problems for consumers downstream, which will have to be updated or they will no longer get their architectures installed.

If we really want to keep the VM image small, we can always remove the executables after installation.

[rhatdan]

You can keep qemu-user-static installing all packages. But if we break it up to only install the aarches we want on CoreOS, it is a win/win.

[bugfood]

The referenced Fedora ticket is now closed:
https://bugzilla.redhat.com/show_bug.cgi?id=2061584

I've been installing qemu-user-static-x86 as a workaround and that has behaved as intended, with one exception (so far):

$ sudo
Error while loading /usr/bin/sudo: Permission denied

This is related to the binfmt_misc configuration. I would have been lost without this post:
https://forums.gentoo.org/viewtopic-p-8674527.html

Specifically for the podman machine, this command can add the necessary "C" flag to a running rootful VM:

podman machine ssh 'echo -1 > /proc/sys/fs/binfmt_misc/status && for i in /usr/lib/binfmt.d/qemu-* ; do sed -r "s/:F\$/:CF/" "$i" > /proc/sys/fs/binfmt_misc/register || break ; done'

I don't know how to make that persist across reboots (I'm not familiar enough with coreos and don't want to maintain my own package).

Hopefully this info will be useful to someone.

[gbraad]

a good enough reason for not just installing qemu-user-static

I do not think you run containers for Nios, Microblaze. HPPA, Alpha, etc. these aren't necessary for a system. we do care about the disk size as this counts towards redistribution like time to download and get to a working environment.

[shanesmith]

As of today it seems the Podman machine images come with qemu-i386-static and qemu-x86_64-static, which means we no longer need to install qemu-user-static if those are the only archs we're interested in, hooray!

If someone is interested in other archs though, installing qemu-user-static actually fails due to these existing binaries, just a heads up.

Relatedly, is there a repo or changelog or something that I can follow to learn about changes made to the podman machine image?

[rhatdan]

The podman macine image is just fedora coreos. So if you follow the development over there, you can see this information.
qemu-user-static was broken into many sub packages.
qemu-user-static-aarch64 x86_64 2:6.2.0-12.fc36 updates 2.3 M
qemu-user-static-alpha x86_64 2:6.2.0-12.fc36 updates 1.2 M
qemu-user-static-arm x86_64 2:6.2.0-12.fc36 updates 1.9 M
qemu-user-static-cris x86_64 2:6.2.0-12.fc36 updates 1.2 M
qemu-user-static-hexagon x86_64 2:6.2.0-12.fc36 updates 1.5 M
qemu-user-static-hppa x86_64 2:6.2.0-12.fc36 updates 1.2 M
qemu-user-static-m68k x86_64 2:6.2.0-12.fc36 updates 1.3 M
qemu-user-static-microblaze x86_64 2:6.2.0-12.fc36 updates 1.5 M
qemu-user-static-mips x86_64 2:6.2.0-12.fc36 updates 3.3 M
qemu-user-static-nios2 x86_64 2:6.2.0-12.fc36 updates 1.2 M
qemu-user-static-or1k x86_64 2:6.2.0-12.fc36 updates 1.2 M
qemu-user-static-ppc x86_64 2:6.2.0-12.fc36 updates 2.3 M
qemu-user-static-riscv x86_64 2:6.2.0-12.fc36 updates 1.7 M
qemu-user-static-s390x x86_64 2:6.2.0-12.fc36 updates 1.3 M
qemu-user-static-sh4 x86_64 2:6.2.0-12.fc36 updates 1.5 M
qemu-user-static-sparc x86_64 2:6.2.0-12.fc36 updates 1.8 M
qemu-user-static-x86 x86_64 2:6.2.0-12.fc36 updates 1.8 M
qemu-user-static-xtensa x86_64 2:6.2.0-12.fc36 updates 2.0 M

Each of these can be installed independently. I am not sure why installing a subpackage should block installing the group, that might be a bug. Anyways you can install the arch that you need by directly installing one of the above packages.

[shanesmith]

Oh I see now with https://getfedora.org/en/coreos?stream=next that as of 36.20220618.3.1 the qemu-user-static-x86 is part of aarch64 builds. I thought Podman was using a special build of coreos, thanks for clearing that up!

[gbraad]

I have not seen this issue. I even did an upgrade from regular F35 to F36. Perhaps specific to coreos?

Anyways. @rhatdan we are trying to prepackage this for M1's release of CRC. Will provide feedback.

[rhatdan]

Fedora CoreOS is useing limited static packages now for emulation.

[rhatdan]

Yes qemu-user-x86* should be installed on the fedora-coreos image that is used on M1s.
Similarly qemu-user-ARM* should be installed fedora-coreos image used on X86 machines.

[gbraad]

qemu-user-x86*

Would you also use the qemu-user-static-x86 (32bit)? I only tested with qemu-user-static-x864_64 (64bits) and this worked as expected.

And it would be qemu-user-static-aarch64 for the x86_64. As the qemu-user-static-arm is pre-v8, like v7 and this is might also not be as helpful. it is questionable if people would use this. I do not see many requests of people on Intel wanting to run their raspi image ;-).

[dustymabe]

Yes qemu-user-x86* should be installed on the fedora-coreos image that is used on M1s.

Correct. But to be specific: We install the qemu-user-static-x86 package. No others.

Similarly qemu-user-ARM* should be installed fedora-coreos image used on X86 machines.

Incorrect. See coreos/fedora-coreos-tracker#1237

[rhatdan]

Yes I was just indicating that X86 processes can be emulated on Arm Systems and vice versa.

[gbraad]

We might decide against that for the x86_64 images of CRC. It downloads about 12-16MB but still unpacks to 200+ MBs. Will get back to this.

For WSL2 I didn't get this to easily work as systemd is not full functional, so an Windows ARM didn't work as expected yet.

[ssbarnea]

@gbraad ping me about anything arm64 related, I had decent experience with macos/fedora/ubuntu/windows-11 all on arm64(M1), with the only note that it was zero WSL support because M1 does not allow nested virtualization and I did not want to repartition my mac to test if WSL works on arm or or not. I wonder if they addressed this on M1 or not yet.

[gbraad]

Thanks, much appreciated. We are in the process to enable binfmt in the podman bundles for CRC that are for the M1, as this uses vfkit instead of Qemu as the main hypervisor.

[crepererum]

I've encountered this issue today while pulling https://hub.docker.com/r/serjs/go-socks5-proxy/tags which has a somewhat broken latest tag at the moment:

On an x64 Linux laptop podman run pulls the image and then exists due to the binary format mismatch:

❯ podman run -it --rm serjs/go-socks5-proxy@sha256:97fb702ee08ba693da94a190caef329d41ca9fd4753408d3e89a38aff56d826d /socks5
Resolving "serjs/go-socks5-proxy" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/serjs/go-socks5-proxy@sha256:97fb702ee08ba693da94a190caef329d41ca9fd4753408d3e89a38aff56d826d...
Getting image source signatures
Copying blob 7539150c1e11 done
Copying blob c922e5ec9cb5 done
Copying config cfb0ddef80 done
Writing manifest to image destination
Storing signatures
WARNING: image platform ({arm linux  [] v6}) does not match the expected platform ({amd64 linux  [] })
{"msg":"exec container process `/socks5`: Exec format error","level":"error","time":"2022-11-08T13:42:27.000056969Z"}

From an UX perspective, this seems a bit weird. Why is podman even pulling the image in the first place? From an naive PoV, I would expect that the pull-process already detects the architecture mismatch.

[vrothberg]

Why is podman even pulling the image in the first place?

Because Podman does not have a choice. serjs/go-socks5-proxy@sha256:97fb702ee08ba693da94a190caef329d41ca9fd4753408d3e89a38aff56d826d is one image, not a manifest list.

[vrothberg]

From an naive PoV, I would expect that the pull-process already detects the architecture mismatch.

That would break compatibility. There are/were many images in the wild that didn't get their platform right.

[crepererum]

Why is podman even pulling the image in the first place?

Because Podman does not have a choice. serjs/go-socks5-proxy@sha256:97fb702ee08ba693da94a190caef329d41ca9fd4753408d3e89a38aff56d826d is one image, not a manifest list.

The same currently happens with podman run -it --rm --pull=always serjs/go-socks5-proxy:latest /socks5 (where I think podman could choose the architecture). But I guess -- as you've said -- it's a fallout of ironing over broken images/manifests.

[tangentsoft]

Just bumping the issue. It's April 2023, and Podman Desktop 0.13.0 still ships with a version of the Fedora CoreOS (37) where you have to SSH in and install these packages by hand after it runs podman machine for you. Not to sound entitled, but what's the hold-up? The solution is known and available, but half a year later, it still isn't shipping.

My immediate use case is the opposite of M1 support, but the solution's the same. I have an oddball ARMv7 system that supports OCI containers, and rather than dedicate a Raspberry Pi 3 to the project as a headless build host, I'd much rather build on my x86_64 macOS box. I didn't expect to be sent on a major yak shaving expedition merely to get this simple test case working:

 $ podman run -it --rm --platform linux/arm/v7 alpine:latest uname -m

Mind, I've got a solution thanks to the information above. I'm just surprised to see that it isn't in Podman Desktop by this point.

[afbjorklund]

I think arm32 is getting more and more of a niche case (for example it was just excluded from the Kubernetes prebuilt binaries), but I think it would be possible to add the arm architecture to the "usual" amd64 and arm64 ?

[tangentsoft]

While I don't doubt that ARM 32-bit is dwindling in popularity and use, it hasn't fallen off a cliff. The older 32-bit device I'm testing Podman's cross-compilation features with is still being produced, and the same company released a pair of new 32-bit ARM based products in February 2022. These are network switches, so they simply don't need a lot of CPU; in a proper setup, nearly all the processing is handled by the dedicated switch ASIC.

[afbjorklund]

I think all that is needed is to add the qemu-user-static-arm package, to those other two qemu-user packages ?
But according to the information above, Fedora CoreOS only installs support for amd64 (so no arm64 either).

#12899 (reply in thread)
#16451 (comment)

"We'll leave the installation of the emulators up to the user for now."

sudo rpm-ostree install qemu-user-static-arm
sudo reboot

[tangentsoft]

Yes, as I said, the solution does work. The problem is that it's a manual step after each podman-machine rebuild, and finding this solution is 98% of the work, since the symptom — /bin/sh: Exec format error` — turns up way too much irrelevant stuff in a web search. Developers who've spent the last half-decade on Docker will have their expectations violated on this. Podman bills itself as a drop-in replacement for Docker, but on this point, it simply is not.

[tangentsoft]

Sometime in the last year, this has been resolved. I don't know if it happened down at the CoreOS layer, or in podman machine, or even way up in Podman Desktop, but regardless, my uname on armv7l test above works out of the box on a fresh Podman machine.

Whoever did this, thank you!