nixpkgs podman and qemu on darwin don't play well together

[zaphar]

Describe the bug

On darwin podman machine requires qemu to be installed. It also assumes that qemu has either been installed from source or via brew. This results in error messages where podman can't find the edk2 file.

Steps To Reproduce

Steps to reproduce the behavior:

1. install both podman and qemu using nix.
2. run podman machine init test
3. run podman machine start test

Expected behavior

To see a successfully started virtual machine. Instead you will see the following:

$> podman machine init test                                                                                                                                                                                                                 ~
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start test
$> podman machine start test                                                                                                                                                                                                                ~
Starting machine "test"
INFO[0000] waiting for clients...
INFO[0000] new connection from  to /var/folders/z4/0zhrndk13bxd8jdrz_51h75c0000gn/T/podman/qemu_test.sock
Waiting for VM ...
qemu-system-aarch64: -drive file=edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on: Could not open 'edk2-aarch64-code.fd': No such file or directory
Error: dial unix /var/folders/z4/0zhrndk13bxd8jdrz_51h75c0000gn/T/podman/test_ready.sock: connect: connection refused
ERRO[0003] cannot receive packets from , disconnecting: cannot read size from socket: EOF

Additional context

For context the issue in podman appears to be this: https://github.com/containers/podman/blob/main/pkg/machine/qemu/options_darwin_arm64.go#L47 . It would be possible to work around this by modifying by modifying the appropriate machine configuration in ~/.config/containers/podman/machine/qemu/${name}.json but that would require there to be a known good location for the edk2 file to live and right now it's only in the store path for qemu and that can change over time. Ideally the quemu share path would be put in the profile and podman would be patched to search the current nixos profile path for qemu instead.

As it is if you want to use podman and qemu on darwin then qemu can't be installed via nix it would have to installed via brew intead.

Notify maintainers

@zowoq @marsam

Metadata

nix-shell -p nix-info --run "nix-info -m"                                                                                                                                                                                                ~
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.3.0, macOS 12.2.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.7.0`
 - channels(root): `"nixpkgs"`
 - channels(zaphar): `"darwin"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

[zaphar]

This is sort of a bug and sort of not. podman is still useable if you manage the qemu machines yourself on darwin or connect to remote machines. However, I think most podman users will expect the podman machine incantantions` to work.

[zowoq]

nixpkgs podman has lots of problems on darwin (and linux). I mentioned this specific bug to upstream while reporting another issue. containers/podman#13394 (comment)

Also see #163015.

[zaphar]

Interesting. It would be nice if there was a way for us to tell podman where qemu stuff was so it didn't try to guess from a few hard coded paths but I'm not confident that upstream will be doing so anytime soon. I guess I'll have to get something done via an overlay but that kind of sucks.

[ivankovnatsky]

You've probably seen it, just to link it: containers/podman#12379 (comment).

[ethinx]

By following the containers/podman#12379 (comment), add more details about the workaround here:

1. ~/.config/containers/containers.conf is toml file and add helper_binaries_dir = ["/Users/ethinx/.nix-profile/bin"] directly

[engine]
  helper_binaries_dir = ["/Users/ethinx/.nix-profile/bin"]

2. init the podman machine

podman machine init

3. Then the file ~/.config/containers/podman/machine/qemu/podman-machine-default.json will be created and modify as the comment

change the edk2-aarch64-core.fd

"file=edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on"

to abspath in /nix/store

"file=/nix/store/7rw7cya4jac4jkzb6h4p76gzhbq3jjl3-qemu-7.0.0/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on"

Now you could start the machine

podman machine start

[ivankovnatsky]

I just corrected the path for: edk2-aarch64-core.fd, nothing else.

[zaphar]

That is a manual workaround yes. But it sort of defeats the purpose of the using nix if you have to do this and you'll have to update the path if you ever upgrade qemu so it's only a temporary fix.

…

On Tue, May 10, 2022 at 9:11 AM Ivan Kovnatsky ***@***.***> wrote: I just corrected the path for: edk2-aarch64-core.fd, nothing else. — Reply to this email directly, view it on GitHub <#169118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAJ5N2RB22VHLWO6OLOLNTVJJOATANCNFSM5TUPLN6Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Jeremy Wall http://jeremy.marzhillstudios.com ***@***.***

[Atemu]

You can also make the path be ~/.nix-profile/share/qemu/edk2-aarch64-code.fd.

I've found podman to not be a good experience though and would highly recommend giving colima a shot.

[freezeboy]

containers/podman#14324

[retrry]

It seem that this is fixed? Just tried newest version of podman and it correctly created and started VM.

[n8henrie]

I think this will be fixed by containers/podman#17027 in podman 4.5.0 and may have been cherry-picked back to 4.4 in containers/podman@a1cc373

Currently I'm getting podman 4.3 in unstable, it looks like 4.4 is in master: 03d50d8

That said, it seems? to be working on 4.3 right now, so maybe my PR wasn't needed in the first place?

M1 Mac, 13.3

$ nix shell -i -k HOME nixpkgs#bash nixpkgs#podman nixpkgs#qemu --command bash -c 'export PATH=/usr/bin:$PATH; podman machine init && podman machine start'
Downloading VM image: fedora-coreos-37.20230401.2.0-qemu.aarch64.qcow2.xz: done
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

        podman machine start

Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users/n8henrie:/Users/n8henrie

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

        podman machine set --rootful

API forwarding listening on: /Users/n8henrie/.local/share/containers/podman/machine/podman-machine-default/podman.sock

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

        sudo /nix/store/rw74mlcsgf6bq31i0mhvbdgvfhz81cmw-podman-4.3.1/bin/podman-mac-helper install
        podman machine stop; podman machine start

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

        export DOCKER_HOST='unix:///Users/n8henrie/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Machine "podman-machine-default" started successfully

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/out-share-linked-with-nix-profile-install-but-not-otherwise/27561/1

[LoganBarnett]

I just did a recent installation of podman (v4.8.1) and qemu on a fresh setup. I was able to create the VM and use it without issue. I suggest closing this issue :) Thanks for the work here!

My script + invocations used to stand up the VM and use it, in case it matters:

with-podman.sh:

#!/usr/bin/env bash

set -euo pipefail

container_name="nix-run"
script="$@"

podman machine init --cpus 12 --memory 8192 --disk-size 50 \
      --volume $HOME:$HOME || true
podman machine start || true
podman container ls -a | grep $container_name > /dev/null || \
        podman create -t --name $container_name -w /workdir \
            -v $PWD:/workdir nixos/nix
container_id=$(podman start $container_name)
echo "$container_id"
podman exec $container_id $script
podman stop $container_name || true
podman machine stop

And an example run:

$ ./with-podman.sh ls -al
Error: podman-machine-default: VM already exists
Error: cannot start VM podman-machine-default: VM already running or starting
nix-run
total 12
drwxr-xr-x 11 root nobody  352 Dec 24 13:17 .
dr-xr-xr-x  1 root root     77 Dec 24 13:26 ..
drwxr-xr-x 10 root nobody  320 Dec 24 13:41 .git
-rw-r--r--  1 root nobody  283 Dec 22 00:50 README.org
-rw-r--r--  1 root nobody 3809 Dec 24 00:29 flake.nix
drwxr-xr-x  3 root nobody   96 Dec 24 00:30 hosts
drwxr-xr-x  4 root nobody  128 Dec 24 00:40 modules
drwxr-xr-x  3 root nobody   96 Dec 24 00:42 pkgs
drwxr-xr-x  3 root nobody   96 Dec 24 00:52 profiles
drwxr-xr-x  4 root nobody  128 Dec 24 00:41 shell
-rwxr-xr-x  1 root nobody  523 Dec 24 13:40 with-podman.sh
nix-run
Waiting for VM to exit...
Machine "podman-machine-default" stopped successfully

[gnuion]

You can also make the path be ~/.nix-profile/share/qemu/edk2-aarch64-code.fd.

I've found podman to not be a good experience though and would highly recommend giving colima a shot.

why didn't i know about colima sooner? I've actively searched for docker alternatives.

[LoganBarnett]

I retract my previous claim that this ticket should be closed. I don't know why one of my machines was spared this issue. Another machine using a more recent podman at 4.9.3 ran into this issue and required the manual fix listed above.

[kvz]

I've found podman to not be a good experience though and would highly recommend giving colima a shot.

Added these two:

  home.packages = with pkgs; [
    colima
    docker
  ]

Ran:

$ colima start
$ docker run hello-world
Hello from Docker!

Wow, that is indeed a much better experience than what I've been banging my head against so far just to get a container running on macOS. Just wanted to say hi and thank you @Atemu. I'm learning Nix through home-manager and as I'm scavenging through various GitHub issues this is not the first time I've come across your name providing valuable comments turning my showstoppers into elegant solutions. Much appreciated!

[zaphar]

I believe that Podman 5.0.0 might fix this by allowing Podman to use the builtin virtualization technology on darwin. I've started a PR to upgrade it here: #299302

[msgilligan]

Podman 5.0.0 might fix this by allowing Podman to use the builtin virtualization technology on darwin.

Apparently Podman 5.0.x requires vfkit to be installed in order to use the built-in hypervisor. There is no Nix package for vfkit and the current Nix package for Podman doesn't seem to install it. I was able to get Nix-installed Podman to work by installing vfkit with (gasp) Homebrew.

I opened the following issue to report this information separately: #305868

[ldtjcdrs]

I also did this using the homebrew support in nix darwin:

...
              homebrew = {
                enable = true;
                onActivation.upgrade = true;
                taps = [
                  # https://github.com/crc-org/vfkit
                  "cfergeau/crc"
                ];
                brews = [ "vfkit" ];
...

[msgilligan]

I also did this using the homebrew support in nix Darwin

That's cool that you can do that! But I'm trying to replace Homebrew with Nix...

[msgilligan]

Should I open an issue to request a vfkit package?

[msgilligan]

Should I open an issue to request a vfkit package?

Done: #306179