Add support for providing Root CAs to podman machine init

[ericis]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Related to #14589, our host root CA isn't trusted by the virtual Fedora image. This CA must be installed (no special proxy endpoint setting) in the virtual machine to work properly. The podman machine init command successfully downloads the Fedora image and registers it with WSL. I was able to open a new terminal within the image and manually create the CA file to successfully update packages with yum update. However, podman machine start fails, because initialization never completed.

Would love a way to inject the CA file into the machine during initialization for full automation, or at least a way to "continue" initialization after manually fixing the virtual.

Steps to reproduce the issue:

1. Install Podman 4.1.0 to Windows 10 Enterprise behind a proxy that installs its own Root CA (e.g. Zscaler) and does not require a custom proxy URL

2. Run podman machine init

3. Note that the curl SSL fails with errors, because the Root CA doesn't match what is expected by the proxy

4. Run wsl --list to see that "podman-machine-default" was successfully installed

5. Open a terminal using wsl --distribution podman-machine-default

6. Test that connectivity is failing with yum update

7. Manually create the CA file (e.g. cat <<EOT >> ./my.pem ... EOT, where any file name can be used and the content of the ".pem" CA is pasted into the cat prompt)

8. Move the CA file to Fedora's trust directory sudo mv ./my.pem /etc/pki/ca-trust/source/anchors/

9. Update the CA trust with sudo update-ca-trust

10. Test that connectivity now works properly with yum update

11. Exit the WSL terminal (e.g. exit 0)

12. Attempt to continue the podman machine init command and notice that it fails, because the WSL image is already created

13. Attempt to start podman with the command podman machine start and notice that it fails, because podman initialization never completed

Describe the results you received:

The first initialization...

$ podman machine init
Extracting compressed file
Importing operating system into WSL (this may take 5+ minutes on a new WSL install)...
Installing packages (this will take awhile)...
Failed to set locale, defaulting to C.UTF-8
Fedora 35 - x86_64                                                                      0.0  B/s |   0  B     00:03
Errors during downloading metadata for repository 'fedora':
  - Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [SSL certificate problem: unable to get local issuer certificate]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [SSL certificate problem: unable to get local issuer certificate]
Error: package upgrade on guest OS failed: exit status 1

A second attempt to initialize...

$ podman machine init
Error: podman-machine-default: VM already exists

An attempt to start...

$ podman machine start
Starting machine "podman-machine-default"
/bin/bash: line 1: /root/bootstrap: No such file or directory
Error: WSL bootstrap script failed: exit status 127

Describe the results you expected:

Podman allows to continue initialization from where it left off and starts successfully. Ideally, podman machine init allows a custom CA file to be installed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Cannot connect to Podman.

$ podman --version
podman.exe version 4.1.0

Output of podman info --debug:

$ podman info --debug
Cannot connect to Podman.

Package info (e.g. output of rpm -q podman or apt list podman):

n/a Windows with WSL

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Windows 10 Enterprise

[Luap99]

@n1hility PTAL

[AaronRidoutAE]

Please can I up-vote this item as I too can't install Podman directly on Windows 10 behind the corporate firewall packet inspection using Zscaler (like the OP). I have a tortuous work-around running a VM under oracle VM Virtual Box and then run podman from there - ouch. What I'd like to achieve is a one click in an IDE on Windows (eclipse or vscode etc) that uses Podman to build the code

[userid0x0]

Hi,
I am facing an related issue custom proxy URL + self signed certificate. I am evaluating podman 4.1.1 on Windows
Any help is appreciated.

[n1hility]

Hi @userid0x0 @AaronRidoutAE @ericis,

In the upcoming 4.2 release, we are switching to an image with packages pre-installed, which should make adding your missing certs easier (until we have a specific feature for this).

I have a pre-built early version here if you want to try it:
https://github.com/n1hility/podman-builds/blob/main/podman-v4.2.0-dev.msi?raw=true

Using this release you should be able to do a podman machine init, and then after its created you can access it with wsl (or podman machine ssh if started).

At that point you can add the cert + any other changes you need and restart.

sha256sum:
b1ceb7f48d6f5abd1da2149daf16718e9774ed5b85dbfadf85a5f3d9507f121f podman-v4.2.0-dev.msi

[userid0x0]

Hi @n1hility ,
thanks for the effort. I want to share my status

1. In installed podman-v4.2.0-dev -> works fine
2. Using PowerShell I run these commands

PS> $env:http_proxy="proxy.<>.com:8080"
PS> $env:https_proxy="proxy.<>.com:8080"
PS> podman machine init

-> works fine
3. PS> podman machine start -> no errors but the machine doesn't run (according to podman machine list)
4. Using wsl -d podman-machine-default I entered the machine and created a /etc/profile.d/my-proxy.sh

cat << EOF > /etc/profile.d/my-proxy.sh
export http_proxy="http://proxy.<>.com:8080"
export https_proxy="http://proxy.<>.com:8080"
export ftp_proxy="http://proxy.<>.com:8080"
export no_proxy="localhost,127.0.0.1,.<>.com"
EOF
source /etc/profile.d/my-proxy.sh

5. Next I installed our certificate

yum install wget
wget -O /etc/pki/ca-trust/source/anchors/mycert.pem http://<>/cert.crt
update-ca-trust

A test wget <internal page using this cert> succeeds.
6. Again a try

PS> podman machine start
# no error message - Machine "podman-machine-default" started successfully
PS> podman machine list
# shows not running
podman ps 
# Cannot connect to Podman.
# Please verify your connection to the Linux system using podman system connection list, 
# or try podman machine init and podman machine start to manage a new Linux VM

7. Next I checked the WSL version and switch from WSL 1 -> 2 PS> wsl --set-version podman-machine-default 2
8. PS> podman machine start PS> podman ps succeeds now , also PS> podman search busybox
9. Now I modified /etc/containers/registries.conf and added our internal dockerhub to unqualified-search-registries from WSL.
   A PS> podman search <something that only exists on our local dockerhub> responds with

Error: 1 error occurred:
        * couldn't search registry "dockerhub.<>.com": pinging container registry dockerhub.<>.com: Get "https://dockerhub.<>.com/v2/": EOF

If I do the same thing within the WSL podman-machine-default it succeeds. e.g. wsl -d podman-machine-default -- podman search <something that only exists on our local dockerhub>
I am taking the latter as a proof that the certificate is correctly installed.

TL;DR
Much better than before. I do not know if the problem in 9 is a different issue.

[userid0x0]

I took a log, I have no clue what could be the problem - sorry.

podman --log-level trace search <something that only exists on our local dockerhub>
time="2022-07-22T14:25:14+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level trace"
time="2022-07-22T14:25:14+02:00" level=debug msg="Called search.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe --log-level trace search <    >)"
time="2022-07-22T14:25:14+02:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\<>\\\\.ssh\\\\podman-machine-default\" SHA256:<> ssh-ed25519"
time="2022-07-22T14:25:14+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.2.0/libpod/_ping"
time="2022-07-22T14:25:14+02:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2022-07-22T14:25:14+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.2.0/libpod/images/search"
Error: 1 error occurred:
        * couldn't search registry "dockerhub.<>.com": pinging container registry dockerhub.<>.com: Get "https://dockerhub.<>.com/v2/": EOF

[n1hility]

@userid0x0 ah yeah, installing as wsl V1 will lead to issues. In a future update, I aim to prevent that. BTW You should not need to do step 4. The proxy settings are picked up on every start now (if set they will be used, if not set they will be cleared)

Regarding the registry changes, did you try bouncing the machine (podman machine stop, podman machine start)? Remote requests are handled by a systemd service so the changes to configs can potentially be cached across operations.

[userid0x0]

@n1hility Yes, I started/stopped the machine in between - sorry missed that in my description.
Let's see on Monday what happens after a restart of the workstation.
Any idea how I can troubleshoot further? The log seems only to show logs from windows side not what the podman-machine is trying to do.

[userid0x0]

I just want to confirm back that a restart of the workstation did not solve the problem with the internal dockerhub.

[userid0x0]

Wireshark was helpful. Interesting to note: I set no_proxy in step 4) within the WSL machine. The last placeholder is for our internal seachdomain .internal.network.com.
I noticed that when I try to run podman from windows side it tries to access our internal dockerhub though the proxy. I just remember from old days that the case of http_proxy/no_proxy sometimes makes a difference. So I added no_proxy & NO_PROXY and now it works (after bouncing the machine).

TL;DR

cat << EOF > /etc/profile.d/my-proxy.sh
export http_proxy="http://proxy.<>.com:8080"
export HTTP_PROXY=${http_proxy}
export https_proxy="http://proxy.<>.com:8080"
export HTTPS_PROXY=${https_proxy}
export ftp_proxy="http://proxy.<>.com:8080"
export no_proxy="localhost,127.0.0.1,.<>.com"
export NO_PROXY=${no_proxy}
EOF

[rhatdan]

@userid0x0 @n1hility Is this still an issue?

[userid0x0]

For me the things work now. Good work guys! Thanks!
Regarding the case of no_proxy/NO_PROXY it's up to you.

[rorygilfillan-absa]

podman 4.2.0 doesn't install for me

[ericis]

@rhatdan and @n1hility I'll test today as well.

[n1hility]

@rorygilfillan-absa do you have any sort of virus scanner running and did it trigger any alerts during the process?

[ericis]

@n1hility I got the same error. I ran the standard Windows "Unblock", since it was initially blocked from even running because it was "not trusted" (signed).

Steps:

1. Downloaded
2. Run (blocked because of Windows trust issue on unsigned software from the internet)
3. Reviewed your repo and changes 👍🏻
4. Unblock
5. Run again
6. Accept prompt for escalated privileges (expected)
7. Installation begins, but results in the error @rorygilfillan-absa shared above

Windows 10 Enterprise

[n1hility]

Moving the dev build installer issue to a discussion and closing this one since the main limitation is addressed. Feel free to reopen otherwise

[carljmosca]

Not suggesting this needs to be reopened as the combination of getting the cert(s) via curl, copying to /etc/pki/ca-trust/source/anchors/ and then running update-ca-trust worked nicely for me. Just wondering if it's documented somewhere that I have missed.

[n1hility]

@carljmosca Good point. A customized trust setup isn't documented, would you be interested in contributing a section in our windows guide? It's located here:

https://github.com/containers/podman/blob/main/docs/tutorials/podman-for-windows.md

[carljmosca]

Sure, I can take a look but I did this on macOS ;)

[n1hility]

@carljmosca awesome thanks! and oh good point on mac - hmm maybe it should be a separate md file wdyt?

[carljmosca]

yes, I just had the same thought

[n1hility]

Link to installer issue discussion: #15133