[macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

thomasw64 · 2021-06-06T13:08:48Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman machine start and the other podman machine commands use not supported qemu accelerator hvf on aarch64/darwin (MacOS 11.4 on M1).

podman/pkg/machine/qemu/options_darwin_arm64.go adds to the invocation of qemu --accel hvf.

The standard qemu built from brew formula did not contain hvf support, so I rebuilt qemu with configure option --enable-hvf which did not result in the expected outcome. Verifying the binary with
qemu-system-aarch64 -accel help states only tcg supported. I reviewed the qemu source, and as it looks there is no implementation of hvf for aarch64 in qemu yet.

If I read it correctly the qemu code for target arm or aarch64 does not contain any calls to the Hypervisor Framework (hvf).
See https://gitlab.com/qemu-project/qemu/-/tree/master/target/i386/hvf vs https://gitlab.com/qemu-project/qemu/-/tree/master/target/arm

Steps to reproduce the issue:

Create machine

# podman machine init                                                                              
Extracting compressed file

Verify the machine

# podman machine list                                                                                  
NAME                     VM TYPE     CREATED             LAST UP
podman-machine-default*  qemu        About a minute ago  About a minute ago

Start the machine

# podman machine start                                                                                         
Waiting for VM ...
qemu-system-aarch64: -accel hvf: invalid accelerator hvf
Error: dial unix /var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

Describe the results you received:

qemu reports that hvf is an invalid accelerator.

Describe the results you expected:

Machine is started with the right qemu parameters.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens every time.

Output of podman version:

# podman --version                                                                                            
podman version 3.2.0

Output of podman info --debug:

# podman info --debug                                                                                          
Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.0/libpod/_ping": dial unix ///var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman-run--1/podman/podman.sock: connect: no such file or directory

Package info (e.g. output of rpm -q podman or apt list podman):

Output of brew list podman

/opt/homebrew/Cellar/podman/3.2.0/bin/podman
/opt/homebrew/Cellar/podman/3.2.0/etc/bash_completion.d/podman
/opt/homebrew/Cellar/podman/3.2.0/share/fish/vendor_completions.d/podman.fish
/opt/homebrew/Cellar/podman/3.2.0/share/man/ (158 files)
/opt/homebrew/Cellar/podman/3.2.0/share/zsh/site-functions/_podman

Output of brew info podman

podman: stable 3.2.0 (bottled)
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/3.2.0 (167 files, 29.3MB) *
  Poured from bottle on 2021-06-06 at 09:38:46
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
==> Caveats
fish completions have been installed to:
  /opt/homebrew/share/fish/vendor_completions.d
==> Analytics
install: 2,316 (30 days), 9,193 (90 days), 26,804 (365 days)
install-on-request: 2,315 (30 days), 9,190 (90 days), 26,365 (365 days)
build-error: 0 (30 days)

Output of brew list qemu

/opt/homebrew/Cellar/qemu/6.0.0/bin/elf2dmp
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-edid
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-img
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-io
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-nbd
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-storage-daemon
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-aarch64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-alpha
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-arm
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-avr
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-cris
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-hppa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-i386
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-m68k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblaze
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblazeel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64el
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mipsel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-moxie
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-nios2
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-or1k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv32
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-rx
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-s390x
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4eb
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-tricore
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-x86_64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensaeb
/opt/homebrew/Cellar/qemu/6.0.0/share/applications/qemu.desktop
/opt/homebrew/Cellar/qemu/6.0.0/share/icons/ (10 files)
/opt/homebrew/Cellar/qemu/6.0.0/share/qemu/ (107 files)

Output of brew info qemu

qemu: stable 6.0.0 (bottled), HEAD
Emulator for x86 and PowerPC
https://www.qemu.org/
/opt/homebrew/Cellar/qemu/6.0.0 (161 files, 569.4MB) *
  Built from source on 2021-06-06 at 11:15:13
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/qemu.rb
License: GPL-2.0-only
==> Dependencies
Build: libtool ✔, meson ✔, ninja ✔, pkg-config ✔
Required: glib ✔, gnutls ✔, jpeg ✔, libpng ✔, libslirp ✔, libssh ✔, libusb ✔, lzo ✔, ncurses ✔, nettle ✔, pixman ✔, snappy ✔, vde ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 14,340 (30 days), 40,554 (90 days), 122,362 (365 days)
install-on-request: 14,262 (30 days), 40,376 (90 days), 121,152 (365 days)
build-error: 0 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Hardware: Mac Mini M1 16G
ProductName: macOS
ProductVersion: 11.4
BuildVersion: 20F71

The text was updated successfully, but these errors were encountered:

thomasw64 · 2021-06-06T13:30:09Z

I was able to start the machine, by changing -accel hvf to -accel tcg in .config/containers/podman/machine/qemu/podman-machine-default.json

...
  "-device",
  "virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0",
  "-accel",
  "tcg",
  "-cpu",
  "cortex-a57",
...

output of podman machine start

Waiting for VM ...
qemu-system-aarch64: -accel tcg: qemu_mprotect__osdep: mprotect failed: Permission denied

output of podman machine list

NAME                     VM TYPE     CREATED        LAST UP
podman-machine-default*  qemu        2 minutes ago  Currently running

Now, it looks, I found another issue: The coreos image contains (34.20210601.dev.0) podman 3.1.2 instead of 3.2.0.

mheon · 2021-06-06T19:05:07Z

This is expected - we're waiting on a few more patches to merge upstream in Qemu for this to be supported. podman machine is still in heavy development and very much experimental - we hope to have things more polished and ready in the next few releases.

The Podman issue may be my fault, I'll bring it up at scrum on Monday. 3.1.2 and 3.2.0 should be able to talk to each other. We are needing to put out a Podman 3.2.1 already for another bug, so fixing this as the same time is probably not a big deal.

baude · 2021-06-07T21:06:47Z

stay tuned, I will have instructions for you shortly.

github-actions · 2021-07-08T00:03:32Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-07-08T11:49:13Z

@baude any update?

github-actions · 2021-08-08T00:03:35Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-08-09T16:22:14Z

@baude another month, and no merge?

baude · 2021-08-09T17:52:50Z

correct ... i have checked with Sergio but he is now on PTO.

lpil · 2021-09-01T10:28:31Z

With the change in Docker's pricing today this feature becomes a lot more desired! It would be great to get this in the near future. Thank you

baude · 2021-09-01T14:31:34Z

@lpil unfortunately,we are subject to the merging of qemu patches upstream. once that is done, we will have it all set.

Conan-Kudo · 2021-09-01T14:44:26Z

@baude What's the current state of the patchset for QEMU? Is there a link to the latest discussions about them that I can follow?

baude · 2021-09-01T18:44:37Z

no upstream change, the submitter needs to make some changes and resubmit

thomasw64 · 2021-09-02T07:11:38Z

@baude circumventing the QEMU issue temporarily with updating the -accel hvf to -accel tcg is not a good choice? At least people can run podman machine with that on M1 Macs.

thomasw64 · 2021-09-02T10:56:03Z

Before 3.3.1 I could start the machine with fixing the -accel tcg. Now any podman machine command says:

Error: due to missing upstream patches, Apple Silicon is not capable of running Podman machine yet

I think it would be better running with less optimal acceleration than tell you cannot do it at all.

simnalamburt · 2021-09-05T07:02:57Z

FYI, I have patched and packed both qemu and podman due to this issue. You can check this out with the command below:

brew install simnalamburt/x/podman-apple-silicon

# Example
podman machine init
podman machine start
podman run -p 8080:80 docker.io/nginx

References

https://github.com/simnalamburt/homebrew-x/

jonpspri · 2021-09-05T12:44:03Z

@simnalamburt I'm working on a PR with a podman patch for the podman side of the equation. I'll include awareness of /opt/homebrew directories (which I expect will create a separate firestorm) and backwards compatibility with non-HVF qemu.

Having said that, your qemu patches look much farther along than my experiment :). I suppose we'll have to see where the slow-moving upstream review of all that is at.

simnalamburt · 2021-09-05T12:57:03Z

These are the patch that I used.

qemu: https://github.com/simnalamburt/qemu/tree/hvf, it's based on qemu 6.1.0
podman: https://github.com/simnalamburt/podman, it's based on the main branch of podman

simnalamburt · 2021-09-05T13:02:20Z

@simnalamburt I'm working on a PR with a podman patch for the podman side of the equation.

By the way can I take a look of your podman patch? Maybe we can collaborate with it.

My PR is currently at #11449

jonpspri · 2021-09-05T13:10:18Z

Hi @simnalamburt . My podman patch is in https://github.com/jonpspri/podman/tree/homebrew.

I did what I could to make it backwards-compatible. I was working from head rather than 3.3.1, but I don't think that'll make much difference on a diff. Feel free to grab as much as you like and work it into your PR, since you've already started it. I'll go pull and build your qemu and re-test on my hardware.

jonpspri · 2021-09-05T13:32:04Z

@simnalamburt I looked at #11449 . I think we should add looking for the BIOS image in multiple locations into that PR as well (see my tree); however, to workaround this particular issue we may wish to add a separate PR for having qemu fallback to TCG acceleration if HVF acceleration is not available. That's a one-liner I can set up in a separate PR.

[Note: I created PR #11453 to address the BIOS FD issue, since it's very narrowly focused. The gvproxy conversation seems much more complicated.]

Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirable to leave to embarrass the upstream Qemu into approving the HVF patches. [NO TESTS NEEDED] Signed-off-by: Jonathan Springer <[email protected]> Signed-off-by: Jonathan Springer <[email protected]>

mheon · 2021-09-05T22:52:41Z

I believe @ashley-cui is also maintaining a Qemu build for Homebrew which includes the appropriate patches to run on M1.

Qemu TCG Accel fallback for Apple Silicon. Iss #10577

jonpspri · 2021-09-08T11:09:26Z

Patch #11451 worked around this by allowing qemu to fall back onto TCG (software) acceleration. I don't know whether we want to keep the Issue open to track upstream progress? @baude , is there a label for that?

simnalamburt · 2021-09-09T16:39:14Z

With #11451 being merged, I think there're nothing more we should do in the podman side for the podman's M1 support except #11449 and #11480.

hurricanehrndz · 2021-09-15T12:00:40Z

@simnalamburt

Did you use Alexander's patch for hvf support?
https://patchew.org/search?q=hvf%3A+Implement+Apple+Silicon+Support

Looks like there is a recent update to the patch, changing how qemu would be called:

To use hvf support, please make sure to run -M virt,highmem=off to fit
in M1's physical address space limits and use -cpu host.

simnalamburt · 2021-09-15T13:22:17Z

@simnalamburt

Did you use Alexander's patch for hvf support?
https://patchew.org/search?q=hvf%3A+Implement+Apple+Silicon+Support

Looks like there is a recent update to the patch, changing how qemu would be called:
To use hvf support, please make sure to run -M virt,highmem=off to fit
in M1's physical address space limits and use -cpu host.

Yes I used that patch and podman is already using -M virt,highmem=off and -cpu option.

Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirable to leave to embarrass the upstream Qemu into approving the HVF patches. [NO TESTS NEEDED] Signed-off-by: Jonathan Springer <[email protected]> Signed-off-by: Jonathan Springer <[email protected]>

daniellaera · 2021-09-30T21:29:08Z

why my config file is not create when I installed podman ?

~/.config/containers/podman/machine/qemu/podman-machine-default.json

is not present

ashley-cui · 2021-10-12T19:37:42Z

Closing as we now have support for M1 via Homebrew's Qemu

fourstepper · 2021-10-13T12:29:47Z

@ashley-cui not sure if this is in the scope of this issue, but should it be possible to run and build linux/amd64 with podman on MacOS via brew's qemu, or is this a feature possibly coming at a later date?

See containers/podman#10577

benfro · 2023-07-08T13:21:14Z

Closing as we now have support for M1 via Homebrew's Qemu

Hi.
I'm desperately trying to get Podman installed via brew to work. Version 4.5.1 does not. Error is "qemu exited unexpectedly with exit code -1, stderr: qemu-system-x86_64: Unknown Error"
In what way is there "support for M1 via Homebrew's Qemu"?

I have been looking into posts for three weeks now and not one single one of thm seem to give a good solution to this issue. As far as I am concerned, there is a big issue with Podman, brew and qemu on M1 machines still.

Regards,

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 6, 2021

thomasw64 mentioned this issue Jun 6, 2021

[macos] Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2" #10578

Closed

github-actions bot added the stale-issue label Jul 8, 2021

rhatdan removed the stale-issue label Jul 8, 2021

github-actions bot added the stale-issue label Aug 8, 2021

rhatdan removed the stale-issue label Aug 9, 2021

jonpspri mentioned this issue Sep 5, 2021

Qemu TCG Accel fallback for Apple Silicon. Iss #10577 #11451

Merged

openshift-merge-robot added a commit that referenced this issue Sep 6, 2021

Merge pull request #11451 from jonpspri/qemu-apple-silicon-hvf-tcg

6848153

Qemu TCG Accel fallback for Apple Silicon. Iss #10577

ashley-cui closed this as completed Oct 12, 2021

judaew added a commit to macports/macports-ports that referenced this issue Dec 4, 2021

podman: podman-machine must have support for M1

2a1e0b3

See containers/podman#10577

PedroRegisPOAR mentioned this issue Oct 3, 2023

The podman statically compiled and QEMU ES-Nix/podman-rootless#15

Open

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 7, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

[macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

thomasw64 commented Jun 6, 2021

thomasw64 commented Jun 6, 2021

mheon commented Jun 6, 2021

baude commented Jun 7, 2021

github-actions bot commented Jul 8, 2021

rhatdan commented Jul 8, 2021

github-actions bot commented Aug 8, 2021

rhatdan commented Aug 9, 2021

baude commented Aug 9, 2021

lpil commented Sep 1, 2021

baude commented Sep 1, 2021

Conan-Kudo commented Sep 1, 2021

baude commented Sep 1, 2021

thomasw64 commented Sep 2, 2021

thomasw64 commented Sep 2, 2021

simnalamburt commented Sep 5, 2021 •

edited

Loading

jonpspri commented Sep 5, 2021 •

edited

Loading

simnalamburt commented Sep 5, 2021

simnalamburt commented Sep 5, 2021

jonpspri commented Sep 5, 2021

jonpspri commented Sep 5, 2021 •

edited

Loading

mheon commented Sep 5, 2021

jonpspri commented Sep 8, 2021

simnalamburt commented Sep 9, 2021

hurricanehrndz commented Sep 15, 2021 •

edited

Loading

simnalamburt commented Sep 15, 2021

daniellaera commented Sep 30, 2021

ashley-cui commented Oct 12, 2021

fourstepper commented Oct 13, 2021

benfro commented Jul 8, 2023 •

edited

Loading

[macos] podman machine start on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

[macos] podman machine start on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

Comments

thomasw64 commented Jun 6, 2021

thomasw64 commented Jun 6, 2021

mheon commented Jun 6, 2021

baude commented Jun 7, 2021

github-actions bot commented Jul 8, 2021

rhatdan commented Jul 8, 2021

github-actions bot commented Aug 8, 2021

rhatdan commented Aug 9, 2021

baude commented Aug 9, 2021

lpil commented Sep 1, 2021

baude commented Sep 1, 2021

Conan-Kudo commented Sep 1, 2021

baude commented Sep 1, 2021

thomasw64 commented Sep 2, 2021

thomasw64 commented Sep 2, 2021

simnalamburt commented Sep 5, 2021 • edited Loading

References

jonpspri commented Sep 5, 2021 • edited Loading

simnalamburt commented Sep 5, 2021

simnalamburt commented Sep 5, 2021

jonpspri commented Sep 5, 2021

jonpspri commented Sep 5, 2021 • edited Loading

mheon commented Sep 5, 2021

jonpspri commented Sep 8, 2021

simnalamburt commented Sep 9, 2021

hurricanehrndz commented Sep 15, 2021 • edited Loading

simnalamburt commented Sep 15, 2021

daniellaera commented Sep 30, 2021

ashley-cui commented Oct 12, 2021

fourstepper commented Oct 13, 2021

benfro commented Jul 8, 2023 • edited Loading

[macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

[macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

simnalamburt commented Sep 5, 2021 •

edited

Loading

jonpspri commented Sep 5, 2021 •

edited

Loading

jonpspri commented Sep 5, 2021 •

edited

Loading

hurricanehrndz commented Sep 15, 2021 •

edited

Loading

benfro commented Jul 8, 2023 •

edited

Loading