podman machine VM fails to start on MacOS due to gvproxy runtime path resolution

[StephenWithPH]

Describe the bug

On MacOS, podman machine manages the Linux VM used by podman. As part of that, the gvproxy binary is used to configure networking. containers/podman#11449 introduced behavior which defers to a containers.conf file at runtime to decide in which paths to search for helper binaries like gvproxy.

From man containers.conf:

helper_binaries_dir=["/usr/libexec/podman", ...]

A is a list of directories which are used to search for helper binaries.

The default paths on Linux are:
- /usr/local/libexec/podman
- /usr/local/lib/podman
- /usr/libexec/podman
- /usr/lib/podman

The default paths on macOS are:
- /usr/local/opt/podman/libexec
- /opt/homebrew/bin
- /opt/homebrew/opt/podman/libexec
- /usr/local/bin

Although the derivation for podman includes gvproxy when on MacOS...

nixpkgs/pkgs/applications/virtualization/podman/wrapper.nix

Lines 22 to 38 in f23731c

	let
	podman = podman-unwrapped;
	binPath = lib.makeBinPath ([ ] ++ lib.optionals stdenv.isLinux [
	runc
	crun
	conmon
	slirp4netns
	fuse-overlayfs
	util-linux
	iptables
	iproute2
	] ++ lib.optionals stdenv.isDarwin [
	gvproxy
	qemu
	xz
	] ++ extraPackages);

... the binary is not located in the paths where podman searches.

This can be fixed by adding the following to $HOME/.config/containers/containers.conf:

[engine]
  helper_binaries_dir = ["/nix/store/856hg1fhkikx4sinc8pd5q87k062zqpq-gvproxy-0.1.0/bin"]

Steps To Reproduce

Steps to reproduce the behavior:

1. (on MacOS) nix-env -iA nixos.podman
2. podman machine init && podman machine start

Expected behavior

podman machine start is able to start the VM without getting an error

Notify maintainers

@adisbladis
@saschagrunert
@vdemeester
@zowoq

Metadata

nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-darwin"`
 - host os: `Darwin 20.6.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.15`
 - channels(): `"darwin, nixpkgs-21.05.2521.7bbca9877ca"`
 - nixpkgs: `/Users//.nix-defexpr/channels/nixpkgs`

{ config, pkgs, ... }:

let
  unstable = import (builtins.fetchTarball https://github.com/nixos/nixpkgs/tarball/a506095ad24de8aeca3db4b058761779a034ed71) { config = config.nixpkgs.config; };
in
{
  environment.systemPackages = with pkgs;
    [
      unstable.podman
    ];

[StephenWithPH]

Commentary for future searchers:

Having fixed the problem above, containers/podman#11837 (comment) gave me the solution to solve:

Error: error preparing container <> for attach: error configuring network namespace for container <>: error adding pod <> to CNI network "podman": unexpected end of JSON input

[zowoq]

We can't do anything to fix this here, would need to happen upstream.

It also has other problems on darwin (e.g. store paths get written to the vm config and break when GCed) so I've been thinking that we should remove the wrapper as it isn't working as intended. #141061

[StephenWithPH]

We can't do anything to fix this here, would need to happen upstream.

Upstream had considered (in containers/podman#11449 (comment)) making this build-time configurable. I believe that would have worked quite well for nix. Sadly (for us), they went with runtime.

store paths get written to the vm config and break when GCed

Am I correct to understand that you're referring to what ends up in $HOME/.config/containers/podman/machine/qemu/<VM NAME HERE>.json:

{
  "CPUs": 2,
  "CmdLine": [
    "/nix/store/hbz1lflkjc4kjjbx2gp5kh0dmii1f5n8-qemu-6.1.0/bin/qemu-system-x86_64",
    "-m"

If so, it seems that containers.conf is really the challenge for nix-ifying podman. I stumbled upon containers/common#346, which seems very close to the same set of problems.

I'm going to drop some links among the relevant issues; perhaps there's a way to handle this upstream.

[zowoq]

... build-time configurable. I believe that would have worked quite well for nix. Sadly (for us), they went with runtime.

Buildtime or runtime via a envvar/flag that we could wrap.

Managing containers.conf on darwin the same as we do on linux doesn't really seem like a viable option as it would make the file read-only and podman machine needs it writeable to set VM config, etc. It would also have to be done via an "external" project, nix-darwin (assuming /etc/containers works on darwin?) or home-manager.

Am I correct to understand that you're referring to what ends up in ...

Yes, I think there may also have been more than one path per VM written to either ~/.config/containers or ~/.local/share/containers.