[Bug]: podman.exe machine init fails on Windows Server 2022 w/ UPD

[userid0x0]

Issue Description

Hi,
I tried to use podman for Windows 4.3.1 on Windows Server 2022. Unfortunately the podman.exe machine init fails with an error message

Error: could not create ssh keys: open C:\Users\???\.ssh\podman-machine-default.pub: The system cannot find the file specified.

I managed to isolate the rootcause for the problem already. The machine init fails when creating and storing the SSH private+public key

• podman/pkg/machine/wsl/machine.go

  Line 392 in ce504bb

  if err = createKeys(v, dist, sshDir); err != nil {

• podman/pkg/machine/wsl/machine.go

  Line 524 in ce504bb

  key, err := wslCreateKeys(sshDir, v.Name, dist)

• podman/pkg/machine/wsl/machine.go

  Line 916 in ce504bb

  func wslCreateKeys(sshDir string, name string, dist string) (string, error) {

• podman/pkg/machine/keys.go

  Line 36 in ce504bb

  func CreateSSHKeysPrefix(dir string, file string, passThru bool, skipExisting bool, prefix ...string) (string, error) {

• podman/pkg/machine/keys.go

  Line 77 in ce504bb

  func generatekeysPrefix(dir string, file string, passThru bool, prefix ...string) error {

As far as I can say the implementation expects that if the command wsl is invoked on Windows side the Linux shell is in the same working directory (e.g. /mnt/c/Users/???/.ssh) as Windows. This fails for Windows Server 2022 as the User folders are directory junctions (kind of bind mounted .vhdx volumes) and are inaccessible from WSL side.

C:\Users>dir c:\Users
 Volume in drive C has no label.
 Volume Serial Number is ????-????
 Directory of c:\Users
23/01/2023  17:29    <DIR>          .
??/??/????  ??:??    <DIR>          Administrator.???
...
23/01/2023  ??:??    <JUNCTION>     ame???? [\??\Volume{????????-????-????-????-????????????}\]
....

C:\Users>wsl -d podman-machine-default ls -l /mnt/c/Users/

ls: /mnt/c/Users/ame????: Permission denied
ls: cannot read symbolic link '/mnt/c/Users/ame????': Permission denied
total 0
dr-xr-xr-x 1 user user 512 Nov 10 11:59  Administrator.???
...
lrwxrwxrwx 1 user user   0 Jan 23 ??:??  ame????

With other words the SSH keys don't get placed in C:\Users\???\.ssh while podman machine init as the working directory C:\Users\???\.ssh is not accessible from WSL side.

WORKAROUND
I checked the implementation and noticed that the SSH keys don't get generated IF they already exist. Thus I did

1. I run ssh-keygen -N -t ed25519 -f podman-machine-default on a separate Linux machine
2. copy the resulting podman-machine-default private key podman-machine-default.pub public key to C:\Users\???\.ssh on Windows Server 2022
3. then I run podman.exe machine init - this time it succeeds

May be you can do the private/public key exchange from WSL -> Windows not using the working directory approach but using an PIPE as you do with all the other files podman machine init creates within the WSL.

Steps to reproduce the issue

Steps to reproduce the issue

1. Install podman for Windows 4.3.1 from here
2. open up a cmd view
3. run podman.exe machine init

Describe the results you received

C:\Users\???\>podman --log-level trace machine init
time="2023-01-25T08:49:57+01:00" level=info msg="podman filtering at log level trace"
Extracting compressed file
Importing operating system into WSL (this may take a few minutes on a new WSL install)...
time="2023-01-25T08:50:14+01:00" level=debug msg="Running command: wsl [--import podman-machine-default C:\\Users\\???\\.local\\share\\containers\\podman\\machine\\wsl\\wsldist\\podman-machine-default C:\\Users\\???\\.local\\share\\containers\\podman\\machine\\wsl\\podman-machine-default_fedora-podman-v36.0.115.tar --version 2]"
time="2023-01-25T08:50:23+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default rpm -q --restore shadow-utils 2>/dev/null]"
Configuring system...
time="2023-01-25T08:50:27+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c grep -q Port\\ 59244 /etc/ssh/sshd_config || echo Port 59244 >> /etc/ssh/sshd_config]"
time="2023-01-25T08:50:27+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat >> /etc/sudoers]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /etc/systemd/system/systemd-sysusers.service.d/override.conf]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /home/user/.config/systemd/user/linger-example.service]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c mkdir -p /var/lib/systemd/linger; touch /var/lib/systemd/linger/user]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh]"
time="2023-01-25T08:50:28+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /etc/containers/containers.conf]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c echo wsl > /etc/containers/podman-machine]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /etc/wsl.conf]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /usr/local/bin/enterns; chmod 755 /usr/local/bin/enterns]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /etc/profile.d/enterns.sh]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /etc/wslmotd]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /root/bootstrap; chmod 755 /root/bootstrap]"
time="2023-01-25T08:50:29+01:00" level=debug msg="Running command: wsl [-u root -d podman-machine-default sh -c cat > /usr/local/bin/proxyinit; chmod 755 /usr/local/bin/proxyinit]"
time="2023-01-25T08:50:30+01:00" level=debug msg="Running wsl cmd [-u root -d podman-machine-default ssh-keygen -N  -t ed25519 -f podman-machine-default] in dir: C:\\Users\\???\\.ssh"
Generating public/private ed25519 key pair.
Your identification has been saved in podman-machine-default
Your public key has been saved in podman-machine-default.pub
The key fingerprint is:
SHA256:??? root@???-server1
The key's randomart image is:
+--[ED25519 256]--+
...
+----[SHA256]-----+
Error: could not create ssh keys: open C:\Users\???\.ssh\podman-machine-default.pub: The system cannot find the file specified.

Describe the results you expected

I expect podman machine init to succeed.

podman info output

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM

I am using podman 4.3.1 for Windows

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

This ticket belongs to the discussion I recently started #17194 .

[arixmkii]

This is interesting topic. Especially because podman machine ssh is calling ssh binary on the host OS: https://github.com/containers/podman/blob/main/pkg/machine/wsl/machine.go#L1431 I'm not that Windows expert, but my guess is if there is SSH client on the host, then there should be ssh-keygen.exe as well. At least this assumption I did and still consider valid developing QEMU backend for Podman Machine on Windows. So, switching to windows ssh-keygen.exe could just work.

May be there is a valid reason to use ssh-keygen only inside VM.

[userid0x0]

Hi,
interestingly I discovered the same in the meanwhile. Windows has a ssh-keygen.exe. I wrote now a batch job that installs podman on windows and generates the keys using the windows side ssh-keygen.exe .

@echo off
SET CURDIR=%~dp0
WHERE podman.exe >nul 2>nul
IF %ERRORLEVEL% NEQ 0 (
  echo ERROR: podman.exe not installed
  GOTO :EOF
)

WHERE ssh-keygen.exe >nul 2>nul
IF %ERRORLEVEL% NEQ 0 (
  echo ERROR: ssh-keygen.exe not installed
  GOTO :EOF
)

echo Cleanup podman and WSL
podman machine rm --force >nul 2>&1
wsl --unregister podman-machine-default >nul 2>&1

MKDIR "%systemdrive%%homepath%\.ssh"
CD /D "%systemdrive%%homepath%\.ssh"
ssh-keygen.exe -N "" -t ed25519 -f podman-machine-default
CD /D "%CURDIR%"

IF NOT EXIST "%systemdrive%%homepath%\.ssh\podman-machine-default.pub" (
  echo ERROR: File %systemdrive%%homepath%\.ssh\podman-machine-default.pub missing
  GOTO :EOF
)
IF NOT EXIST "%systemdrive%%homepath%\.ssh\podman-machine-default" (
  echo ERROR: File %systemdrive%%homepath%\.ssh\podman-machine-default missing
  GOTO :EOF
)

REM optinally set the proxy
REM SET HTTP_PROXY=http://proxy.???:8080
REM SET HTTPS_PROXY=http://proxy.???:8080
podman machine init

[arixmkii]

And if the SSH client feature of Windows is not used, then there is an option to install "Beta" (don't know why they call it this way) https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.1.0.0p1-Beta OpenSSH client into Program files folder. This could be just listed as prerequisite to have on of these options installed on your host.

[arixmkii]

Probably we still have to wait for the opinions of the team representative what was the reason to choose inside VM app to generate keys.

[arixmkii]

Additional notes, if this change is accepted, then we can share key generation part between WSL and QEMU backends.

[n1hility]

The reason ssh-keygen.exe was not used is just that not all variants and versions of Windows had it available and working.

We can workaround the issue by generating locally and copying it or piping it out of WSL.

@userid0x0 Thanks for sharing. Your workaround makes sense. I updated the title since this issue should only affect systems with RDS setup and they have enabled UPD.

[arixmkii]

The reason ssh-keygen.exe was not used is just that not all variants and versions of Windows had it available and working.

Does this mean that podman machine ssh would fail on these machines as well? Is it possible (using only Windows builtin features) to have working ssh client, but no working ssh-keygen?

Thinking now what would be the best way to handle potentially missing ssh-keygen for QEMU machine on Windows, but this is for the later time.

[userid0x0]

@n1hility Thanks for taking care. I had to look up what RDS and UPD means - I am not the administrator of the machine nor I am an expert of Windows Server. If you want me to test something, let me know.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@n1hility This is in your hands now, as far as I can tell.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.