podman machine init intermittently fails on M1 Mac with exit status 1

[lajoie]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I am working on a script to set up various bits of tooling including podman. When the script runs podman machine init fails with exit status 1 about 95% of the time but will occasionally succeed. When I run the same command in the same shell that the script ran, it succeeds almost every time but will occasionally fail with exit status 1. Running the command with debug logging provides no additional output.

Thinking there might be something odd with the environment in the subshell that the script commands were running in I capture the environment before run podman machine init and compared it to the environment of the shell that ran the script. They were identical other than expected differences in PWD, PS1, and SHLVL.

Podman is installed via homebrew but coreutils is not (i.e., it's not issues #12379 and #12329).

Steps to reproduce the issue:

1. brew install podman

2. podman machine init

Describe the results you received:
Received exit status 1 after Extracting compressed file but prior to Image resized. being displayed.

Describe the results you expected:
For the container engine VM to initialize properly

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 4.0.3

Output of podman info --debug:

no output as container engine VM won't init

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew info podman
podman: stable 4.0.3 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/Users/clajoie/.arm64/homebrew/Cellar/podman/4.0.3 (172 files, 46.2MB) *
  Poured from bottle on 2022-04-14 at 11:54:15
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
Bash completion has been installed to:
  /Users/clajoie/.arm64/homebrew/etc/bash_completion.d

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
M1 Pro Macbook running MacOS 12.3.1

Here is the related snippet of the tooling setup script

arch=".arm64"
brew_home=${HOME}/.${arch}/homebrew
podman_home=${HOME}/.${arch}/podman

mkdir -p ${podman_home}
mkdir -p ${podman_home}/storage
mkdir -p ${podman_home}/var

echo "export XDG_CONFIG_HOME=${podman_home}" > ${podman_home}/bash_profile
echo "export XDG_DATA_HOME=${podman_home}/storage" >> ${podman_home}/bash_profile
echo "export XDG_RUNTIME_DIR=${podman_home}/run" >> ${podman_home}/bash_profile
echo "alias docker=podman"  >> ${podman_home}/bash_profile

if ! grep -q "podman" "${HOME}/.${arch}/bash_profile"; then
    echo "source ${podman_home}/bash_profile" >> ${HOME}/.${arch}/bash_profile
fi
source ${podman_home}/bash_profile

podman_cpus=${PODMAN_CPUS:-4}
podman_mem=${PODMAN_MEM_MB:-12288}
podman_disk=${PODMAN_DISK_GB:-150}
${brew_home}/bin/podman machine init --cpus ${podman_cpus} --memory ${podman_mem} --disk-size ${podman_disk}
if [ ! $? == "0" ]; then
    echo "Unable to initialize podman"
    exit 1
fi

[lajoie]

Okay, I had some time to dig into this further this morning. I added some printf's throughout the init command and zero'ed in on the SSH key generation as the command that was failing. Specifically here.

I looked at the command that gets invoked:

/usr/bin/ssh-keygen -N "" -t ed25519 -f /Users/clajoie/.ssh/podman-machine-default

I can't see anything odd with that command and that command runs fine when run directly in the shell.

The other thing I noticed is that in the keys.CreateSSHKeys there is no check to see if the key already exists. I think this means that if there were a key there already you'd get an error from the above command.

[mheon]

@baude @jwhonce PTAL

[lajoie]

Is there any documentation on the "ignition file"? It looks like if that is provided the key generation is skipped. That might provide me a means to work around this issue until it gets tracked down.

[baude]

https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine-init.1.md#--ignition-path

[lajoie]

@baude is there documentation on the ignition file itself? I saw the cli arg to pass it in but I can't find any documentation on the actual content of that file.

[baude]

no there is not ... we dont envision people messing with an individual ignition file. But we added support for providing one.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mrexodia]

For me this happened after reinstalling podman. The solution was to run:

podman machine rm
podman machine init

Likely the bug was that ~/.ssh/podman-machine-default already existed from a previous podman installation and the ssh-keygen command would fail.

[rhatdan]

@lajoie @baude @ashley-cui @vrothberg still an issue?

[vrothberg]

I haven't seen it on my M2.

@benoitf have you ?

[benoitf]

no, I don't as well. I think issues are mostly on Windows for 'init' command where WSL is not fully/correctly setup so machine is not imported. But on macOS I don't face issues around the init phase.

[vrothberg]

Thanks! Since the issue is very old and we cannot reproduce anymore, I will close it.

[mrexodia]

Did you uninstall and reinstall podman when trying to reproduce?

[szymczag]

Thanks! Since the issue is very old and we cannot reproduce anymore, I will close it.

@vrothberg - I just had the same issue. Recently I've moved from my Intel-based MacBook to the new one (with M1), and I rsync'ed my ~/, including the ~/.ssh/ directory, with podman-machine-default and podman-machine-default.pub and I had the same issue.

The solution from this comment helps, but I think podman should check it and ask - do you want to use the same key or generate a new one? or smth.

cheers,
maciek