If initializing a machine fails, it should not be added to the list of machines

[benoitf]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

on Windows, if I use the msi and I don't have all requirements (like hyper-v installed, etc)
Calling podman machine init is giving an error but the machine is inserted as a "podman machine"

C:\Users\User>podman machine init
Extracting compressed file
Importing operating system into WSL (this may take a few minutes on a new WSL install)...
Please enable the Virtual Machine Platform Windows feature and ensure virtualization is enabled in the BIOS.
For information please visit https://aka.ms/wsl2-install
Error: the WSL import of guest OS failed: exit status 0xffffffff

and then if I call podman machine list I can see a machine

C:\Users\User>podman machine list
NAME                     VM TYPE     CREATED         LAST UP         CPUS        MEMORY      DISK SIZE
podman-machine-default*  wsl         44 seconds ago  44 seconds ago  0           0B          0B

as it's listed as a stopped machine we can try start

C:\Users\User>podman machine start
Starting machine "podman-machine-default"
There is no distribution with the supplied name.
There is no distribution with the supplied name.
Error: the WSL bootstrap script failed: exit status 0xffffffff

and we have errors as well but if machine is not added it might be better.

And if I reboot after enabling virtualization the machine still fails to start so it's corrupted and should not be there.

C:\Users\User>podman machine start
Starting machine "podman-machine-default"
There is no distribution with the supplied name.
There is no distribution with the supplied name.
Error: the WSL bootstrap script failed: exit status 0xffffffff

Creating a new one works

C:\Users\User>podman machine init foo
Extracting compressed file
Importing operating system into WSL (this may take a few minutes on a new WSL install)...
Configuring system...
Creating mailbox file: No such file or directory
Generating public/private ed25519 key pair.
Your identification has been saved in foo
Your public key has been saved in foo.pub
The key fingerprint is:
....

Steps to reproduce the issue:

Windows 11 without nested virtualization enabled.

1. podman machine init

2. podman machine list

Describe the results you received:
a machine is listed

Describe the results you expected:
no machine as initialization failed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

4.2.0-dev

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

@ashley-cui PTAL

[benoitf]

it seems also that the machine is added to the list of machines before the machine is fully ready.

Machine is becoming listed as a podman machine so in another terminal we can do podman machine start while the message Machine init complete To start your machine run: podman machine start is appearing later.

so we can have errors like

/bin/bash: line 1: /root/bootstrap: No such file or directory
the WSL bootstrap script failed: exit status 127

[ashley-cui]

I think this is because list just goes through the machine directory and unmarshalls machine jsons, if the machine json exists, then podman machine recognizes the machine as "exists". Would the solution here be removing the machine json automatically if the init fails? Something like if the init returns an error, automatically do a rm of the machine?

[rhatdan]

That seems reasonable to me. We don't want to leave around broken configuration.

[benoitf]

@ashley-cui and probably the machine needs to be listed in podman machine list only when the init step completed (and not before)

For example I can launch podman machine init and few seconds later podman machine start in a separate terminal but of course it's failing with random errors as we try to start something that is not yet fully initialized.

[ashley-cui]

So I guess we solve the issue by both having a "ready" state for a machine, as well as removing the config file immediately on a failed init? I'm not sure if we should check for ready status on a podman machine list though, this seems like it could create issues where there are ghost machines that the user doesn't know exists.

[n1hility]

A few of us were discussing this on chat. For the next couple of days my bandwidth is limited and won't be able to work on a patch, but @gbraad might have some cycles to help out if you need that @ashley-cui.

A couple thoughts:

1. It's always possible that a crash or system halt or some other event leave init incomplete, so auto-cleanup is not guaranteed. Additionally, even if the state file is deferred there is a possible conflicting state with the WSL distribution registration and supporting files in the filesystem. Therefore, we should probably have an "incomplete" state. In most cases, the auto-rm would clean this up so the user wouldn't see it, but in the event that didn't work it, it would be clear
2. We should probably have a --keep-on-failure option. Part of the reason it was useful to have our existing behavior, is that it helped with diagnostics. If a user had a problem we were able to direct them to run commands on the WSL instance to pull state

[ashley-cui]

SGTM, will try to take a look, but I don't have a windows box for testing. Will try to get as far as possible with this :)

[benoitf]

I reproduced it easily with a VM having https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ as by default there is no nested virtualization enabled by default on most of virtualization products

But I can also test some custom podman binaries that I can build from source code.

[n1hility]

BTW there was some changes to the Makefile process that make cross-building the msi a little finicky. I can push up a build if needed (or test something as well) without much effort. just ping me

[gbraad]

It is specific to the WSL2 provider as the Init seems to register (writeConfig) before the environment is confirmed to work:

This also happens when you break out during the Importing-stage.

Doing the following:

diff --git a/pkg/machine/wsl/machine.go b/pkg/machine/wsl/machine.go
index 189723ac7..bc0c5b400 100644
--- a/pkg/machine/wsl/machine.go
+++ b/pkg/machine/wsl/machine.go
@@ -349,14 +349,6 @@ func (v *MachineVM) Init(opts machine.InitOptions) (bool, error) {
                return false, err
        }
 
-       if err := v.writeConfig(); err != nil {
-               return false, err
-       }
-
-       if err := setupConnections(v, opts, sshDir); err != nil {
-               return false, err
-       }
-
        dist, err := provisionWSLDist(v)
        if err != nil {
                return false, err
@@ -375,6 +367,14 @@ func (v *MachineVM) Init(opts machine.InitOptions) (bool, error) {
                return false, err
        }
 
+       if err := v.writeConfig(); err != nil {
+               return false, err
+       }
+
+       if err := setupConnections(v, opts, sshDir); err != nil {
+               return false, err
+       }
+
        return true, nil
 }
 

would work

[yosiasz]

this is still an issue for me.

wsl --list shows podman-machine-default
podman machine list shows

NAME                    VM TYPE     CREATED        LAST UP        CPUS        MEMORY      DISK SIZE
podman-machine-default  wsl         2 minutes ago  7 seconds ago  0           0B          0B

running wsl --unregister podman-machine-default removes this vm wsl --list shows no vms

but now I ran podman machine list and I still see this vm lingering around.