Cannot start machine with podman version 4.1.0 in M1 Mac #14303
Comments
|
Experiencing similar(?) issue. M1 Mac. |
|
Getting same issue. If I run in debug mode the following command is run which produces the error if run standalone: /opt/homebrew/bin/qemu-system-aarch64 -m 27000 -smp 10 -fw_cfg name=opt/com.coreos/config,file=/Users/edwardnewman/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix://var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready -device virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0 -accel hvf -accel tcg -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users/edwardnewman,mount_tag=vol0,security_model=mapped-xattr -drive if=virtio,file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2 |
|
Note that it is not possible on MacOS to downgrade to previous podman version as the tap appears to only understand 4.1.0. Was proposing to test to see if that still works as issue appears to be with qemu. |
|
@baude @ashley-cui PTAL |
|
Also running into this, and I saw the same |
|
I am running into the same issue. |
|
Hmm, maybe it looks like something changed in qemu 7.0? Taking a look... |
|
I can confirm that after downgrading the qemu to # =========================
# Versions
% podman --version
podman version 4.1.0
% qemu-system-aarch64 --version
QEMU emulator version 6.2.0
# =========================
# My VMs
% podman machine list
NAME VM TYPE CREATED LAST UP CPUS MEMORY DISK SIZE
opensearch qemu 3 hours ago 3 hours ago 4 34.36GB 107.4GB
podman-machine-default* qemu 3 hours ago 3 hours ago 1 2.147GB 10.74GB
# =========================
# Starting the machine
% podman machine start opensearch
Starting machine "opensearch"
Waiting for VM ...
Mounting volume... /Users/lukas.vlcek:/Users/lukas.vlcek
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:
podman machine set --rootful opensearch
API forwarding listening on: /Users/lukas.vlcek/.local/share/containers/podman/machine/opensearch/podman.sock
The system helper service is not installed; the default Docker API socket
address can\'t be used by podman. If you would like to install it run the
following commands:
sudo /opt/homebrew/Cellar/podman/4.1.0/bin/podman-mac-helper install
podman machine stop opensearch; podman machine start opensearch
You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:
export DOCKER_HOST='unix:///Users/lukas.vlcek/.local/share/containers/podman/machine/opensearch/podman.sock'
Machine "opensearch" started successfully
# =========================
# And finally, SSH into the machine
% podman machine ssh opensearch
Connecting to vm opensearch. To close connection, use `~.` or `exit`
Warning: Permanently added '[localhost]:56046' (ED25519) to the list of known hosts.
Fedora CoreOS 36.20220511.dev.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
Note: But this is not to say that podman 4.1.0 will work correctly with qemu 6.2.0 !!! Now, just a quick notes about how I downgraded the qemu. # At this point I already have podman 4.1.0 installed with qemu 7.0.0
% brew tap-new lukas/local-qemu
% brew extract --version=6.2.0 qemu lukas/local-qemu
% brew uninstall --ignore-dependencies qemu
% brew install lukas/local-qemu/[email protected]
# Done.
# Notice that I did not use the HOMEBREW_NO_AUTO_UPDATE=1 parameter (see the article)
# so the qemu will probably be updated with the next `brew update` command... |
|
@lukas-vlcek I followed the same steps to downgrade (used this article). I think your qemu will not update on the next |
|
In case it's of any use, I'm finding that the max memory I'm able to set on a machine is ✔️ The following works: ✖️ The following results in the |
|
I am also experiencing this issue on an M1 mac running MacOS 12.4 (21F79) with podman 4.1.0 and qemu 7.0.0 (with qemu being installed automatically by homebrew as a dependency of podman). The comment from @shanesmith allowed me to initialize and start a podman machine but I still receive connection failures if I attempt to actually run anything. 🔧 pm-clean util functionI ran this before each of the below tests pm-clean() { podman machine ls -q | sed 's/\*$//' | xargs -L1 podman machine rm --force 2>/dev/null; podman machine ls --noheading; }❌
|
|
@justin-f-perez It's probably a different issue, but I've found that the For this issue, perhaps you could avoid |
|
@shanesmith Thanks for the suggestion! It doesn't seem to matter how I initialize the machine, I get the same meaningless error 😮💨 $ podman machine stop && podman machine rm --force && podman machine init && podman machine start --log-level=trace
INFO[0000] podman filtering at log level trace
DEBU[0000] [/opt/homebrew/bin/qemu-system-aarch64 -m 2048 -smp 1 -fw_cfg name=opt/com.coreos/config,file=/Users/podman-is-so-broken/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix://var/folders/8_/_z_4l9jx6214q4ztktdvhg3d19bx8k/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/8_/_z_4l9jx6214q4ztktdvhg3d19bx8k/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready -device virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0 -accel hvf -accel tcg -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/podman-is-so-broken/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users/podman-is-so-broken,mount_tag=vol0,security_model=mapped-xattr -drive if=virtio,file=/Users/podman-is-so-broken/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2]
Starting machine "podman-machine-default"
DEBU[0000] [/opt/homebrew/bin/qemu-system-aarch64 -m 2048 -smp 1 -fw_cfg name=opt/com.coreos/config,file=/Users/podman-is-so-broken/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix://var/folders/8_/_z_4l9jx6214q4ztktdvhg3d19bx8k/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/8_/_z_4l9jx6214q4ztktdvhg3d19bx8k/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready -device virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0 -accel hvf -accel tcg -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/podman-is-so-broken/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users/podman-is-so-broken,mount_tag=vol0,security_model=mapped-xattr -drive if=virtio,file=/Users/podman-is-so-broken/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2]
Waiting for VM ...
Mounting volume... /Users/podman-is-so-broken:/Users/podman-is-so-broken
DEBU[0013] Executing: ssh [-i /Users/podman-is-so-broken/.ssh/podman-machine-default -p 62930 core@localhost -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -q -- sudo chattr -i / ; sudo mkdir -p /Users/podman-is-so-broken ; sudo chattr +i / ;]
Error: exit status 255
$ echo $?
125 |
|
@justin-f-perez I don't know if this will help you, but I used the information at the bottom of this to configure podman and minikube on my M1. I've been initializing with podman machine init --cpus 6 --memory 12288 --disk-size 50 --image-path next |
|
Hmm, after removing all my VM images and downloading the default, I'm able to reproduce, but before that I was not able to do so. Both times were QEMU 7.0.0, podman 4.1. Taking a further look, but if you know what VM image you're using, let me know. |
|
@ashley-cui what command should be run to see what VM image is being used? @ktram many thanks for the suggestion, but this causes the same error as described in the initial issue description. I also tried just using just ❌
|
|
@justin-f-perez |
[
{
"ConfigPath": {
"Path": "/Users/me/.config/containers/podman/machine/qemu/podman-machine-default.json"
},
"Created": "2022-05-26T15:06:54.843899-05:00",
"Image": {
"IgnitionFilePath": {
"Path": "/Users/me/.config/containers/podman/machine/qemu/podman-machine-default.ign"
},
"ImageStream": "next",
"ImagePath": {
"Path": "/Users/me/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2"
}
},
"LastUp": "0001-01-01T00:00:00Z",
"Name": "podman-machine-default",
"Resources": {
"CPUs": 1,
"DiskSize": 100,
"Memory": 2048
},
"SSHConfig": {
"IdentityPath": "/Users/me/.ssh/podman-machine-default",
"Port": 51642,
"RemoteUsername": "core"
},
"State": ""
}
] |
|
Just ran into this myself as well. On an M1 Macbook Pro. Ended up being the memory limit for me. All I did that caused this was:
After reading through this issue:
|
|
I can also confirm that I have the same issue. |
|
I figured out how to downgrade to 4.0.3 (which is working as expected) and wrote a little helper script note: I uninstalled 4.1.0 first, but unsure whether that's necessary #!/bin/zsh
# shellcheck shell=bash
#
# This script creates a homebrew tap for an old version of podman.
# If no version is provided as the first argument, a list of
# versions is either presented for selection via fzf (if installed)
# or printed to stdout.
set -euo pipefail
list_versions() {
cd "$(brew tap-info homebrew/core | grep '^/' | cut -d' ' -f1)"
git log -p -G 'url.*podman' -- 'Formula/podman.rb' |
grep -e "-.*url.*https://github.com/containers/podman/archive/v.*.tar.gz" | sed 's/^.*v\([0-9.]*\)\.tar.gz.*/\1/'
}
create_tap() {
version="$1"
brew tap-new "$USER/local-podman" ||:
brew extract --version="${version}" podman "$USER/local-podman"
}
version="${1:-}"
if [[ -z "${version}" ]]; then
if command -v fzf; then
echo "Listing versions (this may take a minute)" && sleep 2
version="$(list_versions | fzf)"
else
echo "Usage: ./poldman.zsh <VERSION>"
echo "Example:"
echo "$0 4.0.3"
echo "Listing versions (this may take a minute)"
list_versions
fi
fi
podman machine stop ||:
create_tap "${version}"
brew install "podman@${version}"
brew link "podman@${version}"Hope this helps! |
|
Had the same issue with Qemu from homebrew. Downgraded to 6.2.0_1 and everything is ok. My workaround here Homebrew/homebrew-core#102242 (comment) |
|
Potentially related: https://gitlab.com/qemu-project/qemu/-/issues/903 |
|
I realized today that downgrading to It's not clear why podman 4.1.0 seems to supress the |
|
I'm able to start the machine with memory greater than 3072 with the following changes to
Tried a handful of other cpus listed by This workaround is verified on macOS 12.4 but does result in a crash on macOS 12.3. Have not investigated what might work on these older versions. I really don't know what the implications of these changes are otherwise, any |
|
I've tried running podman with increased memory: I've tried to install Qemu 6.2.0 using: But both options did not resolve my errors. Mac info: MacBook Pro (16-inch, 2021) Edit Run this bash script to migrate your Homebrew from Intel to ARM architecture: Homebrew/discussions#417 (comment) |
|
@shanesmith I have an individual from my customer whom has a 16-inch 2021 Apple M1 Max MacBook Pro on MacOS 12.3.1. He stated that once he made the change discussed here to the QEMU JSON configuration file and started the managed machine caused his MacBook to crash. He said it happened twice. |
|
For me the workaround by changing the memory size has no effect. I am on a MacBook Pro (16-inch, 2021) Apple M1 PRO |
|
@tideman The CPU model has to be changed as well from Perhaps open a new GitHub issue so that you can provide your specific output and configuration? |
|
@cpolizzi Just an observation. I was experiencing the same issue as @tideman . However it feels like there are two different issues in flight. One is the cpu/highmem issue which you have provided a resolution. The second appears to be related to some process that the podman CLI talks to whilst performing operations. I was getting the same "connection refused" error even for podman commands (podman machine list, podman machine rm, podman ps, etc). Not sure when or how the podman macos helper gets involved but it feels like some other process that podman CLI connects to is failing to start and thus socket is not available. Not sure what I did but eventually this was resolved to me but some combination of reinstalling podman/qemu and /or reinstalling helper. |
|
@nycnewman I agree with you that these seem to be two different issues. I have experienced this as @tideman mentioned, albeit extremely rarely. It is why I suggested creating a new issue for it. If I recall correctly I resolved it by wiping the temporary directory. So in the output that @tideman shared I would have done: But, I could be wrong as well. One evening everything was working for this issue and the next day it no longer worked and I received the same results as we are now discussing. It is worth IMHO to open a new issue for this behavior and see what comes. |
|
I'm experiencing what I believe is the same issue. I'm on:
Snippet below: |
|
Upgrade the host to 12.4. |
|
Please upgrade your podman to 4.1.1 and make sure your macos version is 12.4 |
|
After some time debugging, I found the cause of this problem. This problem is caused due to qemu 7.0.0 startup latency (3-5s) that occour in every first qemu execution after Mac Machine machine boots. Podman has some bug that doesn't expect that the creation of socks files, done by the qemu call, can be delayed some seconds, and when podman tries to access the socks files, the qemu is not created them yet, showing the error "Error: dial unix /podman/podman-machine-default_ready.sock: connect: connection refused". To avoid this problem, just execute qemu, even with invalid options (just to initialize), before call "podman machine start". I hope help. |
👋🏽 @ashley-cui and all, thanks for looking into this. I'm able to start the machine (albeit after a long time given I used 2 vCPUs, 2GB of memory and 20GB disk specs for the VM). |
|
@mixpix3ls With your host upgraded to macOS 12.4 and using Podman 4.1.1 and ensuring that the linked fix #14563 is applied to your Podman configuration file for the QEMU |
|
I still have problems getting this to work. First, since I am on Podman v4.1.1 should not What am I missing? I am using nix as package manager if that is of any help
|
|
we are cutting an 4.2 rc1 today. Any chance you couild build that and try? |
|
Unfortunately I don't know how to override and specify this with Nix, but I'll keep an eye out for the new version and see if I get it working when it is out |
|
Here is a snippet from my m1 pro on 4.2 rc1. I hope it is useful. Also, I've use podman machine set --rootful ➜ podman-4.2.0-rc1 podman machine start Waiting for VM ... |
|
Today I had: What worked for me: |
Highly recommend this solution if you are happy to throw away everything by the way... Although I do tend to do "--cpus $(sysctl -n hw.ncpu)", to get all my CPUs in |
Also using nix, and for some reason I was still running into this until 4.2 |
|
EXCELLENT fix from Carlos Eduardo Gorges, on stackoverflow. |
|
This is a problem with virtualizing memory in newer versions of Qemu (7.0+) on aarch64 machines. I resolved the issue by updating the
|
|
Hi folks, I'm also attempting to use podman from within a Nix shell on {
description = "Example Nix Environment";
inputs = {
nixpkgs = { url = "github:NixOS/nixpkgs/nixpkgs-unstable"; };
flake-utils = { url = "github:numtide/flake-utils"; };
};
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
};
# Configure podman
podmanSetupScript = let
registriesConf = pkgs.writeText "registries.conf " ''
unqualified-search-registries = ["docker.io"]
[registries.search]
registries = ['docker.io']
[registries.block]
registries = []
'';
in pkgs.writeScript "podman-setup" ''
#!${pkgs.runtimeShell}
if ! test -f ~/.config/containers/registries.conf; then
install -Dm555 ${registriesConf} ~/.config/containers/registries.conf
fi
'';
in
{
# See: https://gist.github.com/adisbladis/187204cb772800489ee3dac4acdd9947
packages.default = pkgs.mkShell {
buildInputs = with pkgs; [
podman
podman-compose
];
shellHook = ''
# Install required configuration
${podmanSetupScript}
# Start the podman machine. We change the default name to
# VM so we can avoid:
# TODO: This is only necessary in Darwin
# https://github.com/containers/podman/issues/13609#issuecomment-1076366318
podman machine init \
--timezone=local \
--rootful=false \
--log-level=trace \
-v $HOME:$HOME \
vm
'';
};
});
}After entering the shell, and allowing podman to initialise, when I run I've tried adjusting the memory as mentioned by @justkash above, reducing the length of the machine name (down to 'vm'). I've also ensured |
github-actions
bot
added
the
locked - please file new issue/PR
and others
/kind bug
Description
In a M1 Mac, Downloaded the VM via
podman machine initand then when trying to start it viapodman machine startI get thew following error:Tried to download the arch specific VM via
podman machine init --image-path=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220424.3.0/aarch64/fedora-coreos-35.20220424.3.0-qemu.aarch64.qcow2.xzand starting it again with the same error.Steps to reproduce the issue:
podman machine initpodman machine startDescribe the results you received:
Error when trying to start the machine.
Describe the results you expected:
Expected the machine to start successfully.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
The text was updated successfully, but these errors were encountered: