Pass proxy information into Podman Machine VMs

[gclawes]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Podman often has to run behind an http/https proxy, often in corporate environments. This proxy may or may not include SSL inspection capabilities, requiring a trusted SSL CA certificate to be added to a system's trust store.

Minikube solves the first part of this by detecting the standard proxy variables (HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy) and injecting them into the minikube machine at start:

$ minikube start
😄  minikube v1.23.0 on Darwin 11.6.1
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🎉  minikube 1.24.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.24.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🌐  Found network options:
    ▪ HTTP_PROXY=http://proxy.example.com:8080
    ▪ HTTPS_PROXY=http://proxy.example.com:8080
    ▪ NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ http_proxy=http://proxy.example.com:8080
    ▪ https_proxy=http://proxy.example.com:8080
    ▪ no_proxy=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.22.1 on Docker 20.10.8 ...
    ▪ env HTTP_PROXY=http://proxy.example.com:8080
    ▪ env HTTPS_PROXY=http://proxy.example.com:8080
    ▪ env NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

I propose adopting a similar strategy, using the ignition template auto-generated for podman machine, with the additional feature of adding a trusted CA certificate from the openssl standard environment variable SSL_CERT_FILE.

On start, podman machine should detect the presence of these settings and do the following:

1. Inject HTTP_PROXY/etc variables into the machine's systemd environment, so the podman process inside the machine can pull images/make other requests.
2. Copy the file referred to by SSL_CERT_FILE on the host into the podman machine's OS trust store, overriding the built-in single-file trust store certificate.

SSL_CERT_FILE is chosen because it is a well-known environment variable name used to specify a trust store. It could also be used for other SSL requests made by podman outside of the machine instance.

[mheon]

@baude @ashley-cui PTAL

[rhatdan]

@flouthoc PTAL

[ctrought]

+1 for this, I'm sure a lot of corporate users are looking at podman as an option with the docker grace period ending soon.

[flouthoc]

@gclawes @ctrought Could you please try above PR.

[flouthoc]

@gclawes @ctrought Could you please try above PR.

@gclawes @ctrought ^

[ctml91]

@flouthoc , any trick to get this working? I've cloned, run a build for darwin and have done a fresh init, podman server version does report older however so I might have missed a step.

$ env | grep -i proxy
http_proxy=http://proxy.internal.ca:80
https_proxy=http://proxy.internal.ca:80
HTTPS_PROXY=http://proxy.internal.ca:80
HTTP_PROXY=http://proxy.internal.ca:80
NO_PROXY=.internal.ca
$ make podman-remote-darwin
$ ~/podman/bin/darwin/podman version
Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.16.7
Git Commit:   c2f4747fea508a6c6b0fdbf7a51eb6c80ba57f02
Built:        Sun Jan 30 19:08:40 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8

Built:      Wed Dec  8 16:48:10 2021
OS/Arch:    linux/arm64
$ ~/podman/bin/darwin/podman machine init
$ ~/podman/bin/darwin/podman machine start
$ cd ~/mycontainer/
$ ~/podman/bin/darwin/podman build -t mycontainer:latest ./
 <- fails to pull from image
# confirm external site accessible
$ curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

[flouthoc]

@ctml91 Your podman machine is still 3.4.4 i'd suggest replace podman binaries on your system with 4.0.0 or + and do a podman machine rm or podman system reset then run.

I am not sure if macOS contains remote binaries as well or not but on linux i replace both podman and podman-remote binary.

[gclawes]

@flouthoc where is the /usr/bin/podman binary sourced from in podman machine? The only podman binary I have on my machine is the one I built from v4.0.0-rc3, but my machine still runs v3.4.4:

~/.../containers/podman ((v4.0.0-rc3)) $ which -a podman
which: no podman in (/usr/local/opt/file-formula/bin:/usr/local/opt/swagger-codegen@2/bin:/Users/gralaw/bin:/Users/gralaw/.krew/bin:/Users/gralaw/go/bin:/Users/gralaw/.rbenv/shims:/Users/gralaw/.rbenv/bin:/usr/local/opt/python/libexec/bin:/Users/gralaw/.jenv/shims:/usr/local/bin:/usr/local/sbin:/Users/gralaw/bin:/usr/local/opt/coreutils/libexec/gnubin/:/usr/local/opt/gnu-which/libexec/gnubin:/usr/local/opt/gnu-tar/libexec/gnubin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/opt/make/bin:/usr/local/opt/inetutils/libexec/gnubin:/usr/local/opt/gnu-indent/libexec/gnubin:/usr/local/opt/grep/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/ed/libexec/gnubin:/usr/local/opt/file-formula/bin:/usr/local/opt/swagger-codegen@2/bin:/Users/gralaw/bin:/Users/gralaw/.krew/bin:/Users/gralaw/go/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/gralaw/.rbenv/shims:/Users/gralaw/.rbenv/bin:/usr/local/opt/python/libexec/bin:/Users/gralaw/.jenv/shims:/Users/gralaw/bin:/usr/local/opt/coreutils/libexec/gnubin/:/usr/local/opt/gnu-which/libexec/gnubin:/usr/local/opt/gnu-tar/libexec/gnubin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/opt/make/bin:/usr/local/opt/inetutils/libexec/gnubin:/usr/local/opt/gnu-indent/libexec/gnubin:/usr/local/opt/grep/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/ed/libexec/gnubin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/Users/gralaw/.chefvm/bin:/Users/gralaw/.chefvm/bin)
~/.../containers/podman ((v4.0.0-rc3)) $ ./bin/darwin/podman machine init
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start

~/.../containers/podman ((v4.0.0-rc3)) $ ./bin/darwin/podman machine start
Starting machine "podman-machine-default"
INFO[0000] waiting for clients...
INFO[0000] new connection from  to /var/folders/yt/ccjq6fv93sz7d0m_7g26ff240000gn/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
Machine "podman-machine-default" started successfully
~/.../containers/podman ((v4.0.0-rc3)) $ ./bin/darwin/podman version
Client:       Podman Engine
Version:      4.0.0-rc3
API Version:  4.0.0-rc3
Go Version:   go1.17.6
Git Commit:   9b441c0dda57a79dc96eba90be6e2d1ece4b4905
Built:        Wed Feb  2 09:31:38 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8

Built:      Wed Dec  8 16:45:07 2021
OS/Arch:    linux/amd64

I'm able to start a machine with v4.0.0-rc3 but it's still not able to use the proxy vars. I'm getting connections rejected in console log messages:

~/.../containers/podman ((v4.0.0-rc3)) $ ERRO[0241] net.Dial() = dial tcp 38.145.60.21:443: connect: operation timed out
ERRO[0317] net.Dial() = dial tcp 8.43.85.73:443: connect: operation timed out
ERRO[0393] net.Dial() = dial tcp 140.211.169.206:443: connect: operation timed out
ERRO[0469] net.Dial() = dial tcp 209.132.190.2:443: connect: operation timed out

I think this is because ignition sets the variables in /etc/profile.d/proxy-opts.sh, which only makes them available to login shell environments (podman ssh), but I don't think the rest of the system will use it.

I think for this to work correctly it would need to either create a systemd service drop-in with Environment settings at /etc/systemd/system/podman.service.d/proxy.conf (or whatever other container runtime) or a system-wide DefaultEnvironment setting at /etc/systemd/system.conf.d/proxy.conf. I'm leaning towards the second (and remove the profile) so that all processes in the machine get the right proxy settings.