PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment #11941

kishorekkota · 2021-10-12T15:18:52Z

/kind bug

Description

PODMAN running in local MacOS is not able to honor Proxy Env setting in a corporate proxy setting.

Steps to reproduce the issue:

brew install qemu
brew install podman
podman machine init
podman machine start
podman pull nginx:alpine
6

Describe the results you received:

Resolving "nginx" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/library/nginx:alpine...
Error: initializing source docker://nginx:alpine: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.229.227.53:443: i/o timeout

Describe the results you expected:

I am expecting it to be able to pull the image from Docker Library. I am suspecting Image Pull is run at the local VM machine spin up as part of PODMAN machine, if so, how can we get the PROXY setting pushed to the VM. Are there any command for debug in VM ? I am new to PODMAN, sorry if i am asking some of the basic questions.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

○ → podman version
Client:
Version:      3.4.0
API Version:  3.4.0
Go Version:   go1.17.1
Built:        Thu Sep 30 13:44:31 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 15:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.9-200.fc34.x86_64
  linkmode: dynamic
  logDriver: ""
  memFree: 1629442048
  memTotal: 2061860864
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 19m 49.04s
plugins:
  log: null
  network: null
  volume: null
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

DOES NOT APPLY.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes. Does not information on local Podman execution.

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

mheon · 2021-10-12T15:59:58Z

@ashley-cui @baude PTAL

gclawes · 2021-10-12T17:16:05Z

I think you can pass the proxy variables if you set up a custom ignition file, though it would be nice if podman auto-detected proxy settings and passed them through.

Some enterprise proxies also do SSL inspection, which requires loading a custom trusted CA into a user's workstation trust store. I believe tools commonly use the SSL_CERT_FILE env var to do this, similar to HTTP_PROXY/HTTPS_PROXY/etc variables to detect the proxy.

SSL_CERT_FILE is standard in openssl:
https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_default_verify_paths.html

kishorekkota · 2021-10-12T19:21:19Z

@gclawes Can you point me to any readme that can help with the steps and syntax for creating custom ignition file?

bcg62 · 2021-10-12T20:00:24Z

another option would be to create a CA bundle directly from the osx keychain passed into the machine. It's transparent for the end user and less management overhead maintaining multiple bundles.

this is also the approach docker inevitably took.

https://docs.docker.com/desktop/mac/#add-custom-ca-certificates-server-side

kishorekkota · 2021-10-13T12:59:55Z

@bcg62 Issue i am seeing is with connectivity. I could not find details on setting up Ignition file - can someone point me to details on setting up Ignition file ?

ctml91 · 2021-10-21T20:26:14Z

I'm having the same issue, I am not able to use podman on my M1 which is behind a corporate proxy.

STEP 1/23: FROM registry.access.redhat.com/ubi8/ubi-minimal:latest
Trying to pull registry.access.redhat.com/ubi8/ubi-minimal:latest...
Error: error creating build container: initializing source docker://registry.access.redhat.com/ubi8/ubi-minimal:latest: pinging container registry registry.access.redhat.com: Get "https://registry.access.redhat.com/v2/": x509: certificate has expired or is not yet valid

kishorekkota · 2021-10-22T01:57:20Z

@ctml91 Looks like it is getting through Porxy in your case and SSL handshake is failing. Did you do anything specific to set HTTP_PROXY setting ? I have those in my ENV and also tried to mess with container.conf file to see if those are honored by PODMAN. None of attempts get past the connectivity issue.

ctml91 · 2021-10-22T02:08:45Z

@ctml91 Looks like it is getting through Porxy in your case and SSL handshake is failing. Did you do anything specific to set HTTP_PROXY setting ? I have those in my ENV and also tried to mess with container.conf file to see if those are honored by PODMAN. None of attempts get past the connectivity issue.

This is the standard error I get when proxy is not configured. It may look a bit different than your error, but that's probably just dependent on network/proxy setup. I haven't been able to get proxy to work with podman + qemu. Tried copying the ignition config from the default vm stored under ~/.config/containers/podman/machine/qemu/podman-machine-default.ign and adding httpProxy and httpsProxy in the proxy stanza and init a new machine using the modified ign but the vm wouldn't even start. Probably something I've done wrong, but it would be nice for podman to handle this automatically.

$ proxy-on
$ curl https://registry.access.redhat.com > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

$ proxy-off
$ curl https://registry.access.redhat.com > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

pboushy · 2021-10-27T06:55:11Z

This is interesting... podman build appears to have an argument that will pass http proxy envs to the machine, but podman pull doesn't?

Watching in case anyone finds out how to resolve this.

rhatdan · 2021-10-27T09:34:07Z

I think what you are saying is that
podman build --help | grep proxy
--http-proxy pass through HTTP Proxy environment variables (default true)
podman-remote build --help | grep proxy
--http-proxy pass through HTTP Proxy environment variables (default true)

Is happening on the server side not the client side.

pboushy · 2021-10-27T18:59:58Z

@rhatdan sorry about that. Made a critical typo. I've updated it in my original post.

Pulling containers with podman using either podman functionality doesn't work through a proxy using just an argument.
Creating a container appears to.

rhatdan · 2021-10-28T09:51:42Z

This is definitely a bug, and we have opened a discussion on how to fix it.

pboushy · 2021-11-05T23:08:56Z

Some thoughts on how podman can discover the proxy settings:

Ingest the system proxy settings on Mac. You can get the settings by running: scutil --proxy or in Objective-C using the SystemConfiguration framework (I suspect the same is available in Go): https://developer.apple.com/documentation/systemconfiguration/1517088-scdynamicstorecopyproxies
Ingest proxy settings from ENV.
Allow the user to override by adding an argument to the podman commands or configuring a setting somewhere.

Also, PLEASE support no_proxy stuff. Most companies that use proxies need to access stuff that's on the internal network and outside the network, but only outside requires proxy.

znd4 · 2021-11-22T20:32:27Z

The way Docker engine handles this is obnoxious because it requires a GUI toggle and restarting the VM when you jump on / off a corporate network. If there were a way to have podman pull get proxy settings either from the environment or CLI arguments, that'd be sweet

morgans2 · 2021-12-06T12:59:04Z

I'm having the same issue and have attempted several ways of modifying the proxy and cert files. Here the relevant portion of my ignition file. I'd be grateful for anyone who can spot an error.

{"verification":{}}},"proxy":"{"httpProxy": "http://myproxy:80/"}","{"httpsProxy":"http://myproxy:80/","{"noProxy":"somevals"}","security":{"tls":{"certificateAuthorities": [{"source":"https:/site.internal.com/my.pem"}

cpolizzi · 2021-12-18T20:21:10Z

We solved this in a corporate environment on MacOS that requires a secure proxy server to access all external resources. We use the podman managed machine via podman machine. All configuration takes place on the VM. We have been using this successfully now for almost two months now.

Requires: Podman 3.4.0 at a minimum on MacOS host and managed machine.

Configure Proxy Support Globally for All Systemd Services

sudo mkdir /etc/systemd/system.conf.d

Place in /etc/systemd/system.conf.d/10-default-env.conf:

[Manager]
DefaultEnvironment="HTTP_PROXY=<your-proxy-URL>"
DefaultEnvironment="HTTPS_PROXY=<your-proxy-URL>"
DefaultEnvironment="NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

DefaultEnvironment="http_proxy=<your-proxy-URL>"
DefaultEnvironment="https_proxy=<your-proxy-URL>"
DefaultEnvironment="no_proxy=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

Allow Interactive Login Sessions to VM to Use Proxy

Place in /etc/profile.d/proxy.sh:

PROXY_URL=<your-proxy-URL>

export http_proxy="$PROXY_URL"
export https_proxy="$PROXY_URL"
export ftp_proxy="$PROXY_URL"
export no_proxy=".<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

export HTTP_PROXY="$PROXY_URL"
export HTTPS_PROXY="$PROXY_URL"
export FTP_PROXY="$PROXY_URL"
export NO_PROXY=".<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

Configure Trusted Certificates

Place each CA certificate into a separate PEM files in /etc/pki/ca-trust/source/anchors/ then update the certificate authority trust, for example:

install -o root -g root -m 0644 *.pem *.crt /etc/pki/ca-trust/source/anchors
sudo update-ca-trust

Restart Managed Machine

Unfortunately in order to have the Systemd configuration for proxy configuration to take effect you will have to restart the podman managed machine from the host:

podman machine stop
podman machine start

Works on both Catalina and Big Sur.

kishorekkota · 2022-01-03T17:59:25Z

@cpolizzi Thanks a bunch. This worked like a charm.

rhatdan · 2022-01-03T18:02:00Z

Is this still an issue then?

kishorekkota · 2022-01-03T18:02:37Z

Nope. Can be closed ?

ctml91 · 2022-01-03T18:16:58Z

I don't know how feasible it is but it might be nice to have these managed through a set of flags, so one doesn't need to login the VM and update proxy variables as you switch networks (proxied and not)? Or would the preferred solution to be have two VM's spawned by podman, and have one dedicated for the proxied network and another for non proxied?

cdnB8kn · 2022-01-04T15:22:31Z

I would have hoped for passed in parameters (or maybe some other initialization handling/scripting? kind of like a vagrant file? maybe other things are needed beyond proxy?)... Just for a larger dev team, having a set of instructions to configure it every time is... painful (new hires, proxy change, etc). Especially if you can be in a proxy env and non-proxy environment at different times.

gclawes · 2022-01-04T15:29:44Z

Ideally the HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE env vars should be picked up by the podman machine start command and automatically be passed into the ignition file. This is similar to how minikube operates (though I am unsure if it uses ignition):

$ minikube start
😄  minikube v1.23.0 on Darwin 11.6.1
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🎉  minikube 1.24.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.24.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🌐  Found network options:
    ▪ HTTP_PROXY=http://proxy.example.com:8080
    ▪ HTTPS_PROXY=http://proxy.example.com:8080
    ▪ NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ http_proxy=http://proxy.example.com:8080
    ▪ https_proxy=http://proxy.example.com:8080
    ▪ no_proxy=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.22.1 on Docker 20.10.8 ...
    ▪ env HTTP_PROXY=http://proxy.example.com:8080
    ▪ env HTTPS_PROXY=http://proxy.example.com:8080
    ▪ env NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

matt314 · 2022-01-04T16:31:51Z

FWIW, CodeReady containers from RedHat handles this by having config options cover this.

mpiechot@krell 1211 11:26:46 ~/Documents% crc config view | grep proxy | sed 's/secretstuff/g'
- http-proxy                            : http://proxy.external.example.com:80
- https-proxy                           : http://proxy.external.example.com:80
- no-proxy                              : example.com,127.0.0.0/8,localhost,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,.testing

If that's too much to implement, any chance the machine init could create a config file per machine that environment/arbitrary files could be added to ignition?

mheon · 2022-01-04T16:37:47Z

Can you open a feature request issue about this?

gclawes · 2022-01-04T17:36:22Z

Opened: #12739

JdKock · 2022-02-08T09:22:38Z

I'm having the same issue, I am not able to do a podman pull from behind a corporate proxy.
It seems that podman (v3.4.4) is not using my HTTP_PROXY, HTTPS_PROXY, NO_PROXY settings from my MacOS (Monterey).

Is the solution for this to set all these proxy environment settings and certificates manually in the Podman machine?

gclawes · 2022-02-08T17:20:11Z

@JdKock see the discussion in the PR #12739

matt314 · 2022-02-11T13:45:15Z

Is the solution for this to set all these proxy environment settings and certificates manually in the Podman machine?

That's what worked for me. I set the systemd default environment as above in this comment: #11941 (comment)

Only difference (and it's a preference) is I use /etc/environment for the variables instead of a proxy.sh so non-interactive sessions have access to them as well.

mdavis-xyz · 2022-11-03T01:26:45Z

I'm assuming that @cpolizzi 's solution was to modify those files inside the podman VM, not the mac. (Since my Mac doesn't have that first directory, because systemd is not installed.)

I note that by default my proxy env vars were already present in the virtual machine, but all in one line, which I'm guessing is fine.

proxy.sh did not exist. I added that. It didn't resolve my problem.

mdavis-xyz · 2022-11-03T01:31:51Z

In my use case, my proxy is actually CNTLM running on the host (which in turn forwards traffic to my corporate proxy.) I suspect the issue is that there's no path for traffic from the VM to the host? I tried swapping HTTP_PROXY=http://127.0.0.1:3128 for http://localhost:3128. I get the same error.
Which address should I use if my proxy server is on the Mac host?

cpolizzi · 2022-11-03T12:48:39Z

@mdavis-xyz Yes my solution is as follows for the proxy: https://cpolizzi.github.io/podman/podman-as-a-replacement-for-docker-desktop.html#_corporate_network_considerations. This covers both proxying and privately issued certificates. Note that the only real workable solution is to configure SystemD globally for the needed proxy environment variables. If Podman supported / honored SystemD drop in units then configuring SystemD globally would be unnecessary (I also provide an example of what that would look like).

cpolizzi · 2022-11-03T12:50:04Z

@mdavis-xyz By default the podman managed machine cannot make outbound connections from the VM to the host. However, here is how to do it: https://cpolizzi.github.io/podman/orchestrating-podman-vm-to-host-communication.html

mdavis-xyz · 2022-11-04T02:46:14Z

Yes those two instructions together worked for me. Thanks!

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 12, 2021

guillaumerose added the machine label Oct 14, 2021

rhatdan self-assigned this Oct 28, 2021

kishorekkota closed this as completed Jan 3, 2022

esendjer mentioned this issue Feb 12, 2022

ignition: propagate proxy settings from a host into a vm #13209

Merged

praveenkumar mentioned this issue May 10, 2022

[BUG] Podman does not work behind a corporate proxy crc-org/crc#3166

Closed

mkonzal mentioned this issue May 10, 2022

CRC podman does not work behind proxy #14156

Closed

yckbilly1929 mentioned this issue Jan 28, 2023

failed to pull image when use system proxy #14087

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 11, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment #11941

PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment #11941

kishorekkota commented Oct 12, 2021

mheon commented Oct 12, 2021

gclawes commented Oct 12, 2021 •

edited

Loading

kishorekkota commented Oct 12, 2021

bcg62 commented Oct 12, 2021

kishorekkota commented Oct 13, 2021

ctml91 commented Oct 21, 2021

kishorekkota commented Oct 22, 2021

ctml91 commented Oct 22, 2021

pboushy commented Oct 27, 2021 •

edited

Loading

rhatdan commented Oct 27, 2021

pboushy commented Oct 27, 2021

rhatdan commented Oct 28, 2021

pboushy commented Nov 5, 2021 •

edited

Loading

znd4 commented Nov 22, 2021

morgans2 commented Dec 6, 2021

cpolizzi commented Dec 18, 2021

kishorekkota commented Jan 3, 2022

rhatdan commented Jan 3, 2022

kishorekkota commented Jan 3, 2022

ctml91 commented Jan 3, 2022

cdnB8kn commented Jan 4, 2022

gclawes commented Jan 4, 2022 •

edited

Loading

matt314 commented Jan 4, 2022

mheon commented Jan 4, 2022

gclawes commented Jan 4, 2022

JdKock commented Feb 8, 2022 •

edited

Loading

gclawes commented Feb 8, 2022

matt314 commented Feb 11, 2022

mdavis-xyz commented Nov 3, 2022

mdavis-xyz commented Nov 3, 2022

cpolizzi commented Nov 3, 2022

cpolizzi commented Nov 3, 2022

mdavis-xyz commented Nov 4, 2022

PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment #11941

PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment #11941

Comments

kishorekkota commented Oct 12, 2021

mheon commented Oct 12, 2021

gclawes commented Oct 12, 2021 • edited Loading

kishorekkota commented Oct 12, 2021

bcg62 commented Oct 12, 2021

kishorekkota commented Oct 13, 2021

ctml91 commented Oct 21, 2021

kishorekkota commented Oct 22, 2021

ctml91 commented Oct 22, 2021

pboushy commented Oct 27, 2021 • edited Loading

rhatdan commented Oct 27, 2021

pboushy commented Oct 27, 2021

rhatdan commented Oct 28, 2021

pboushy commented Nov 5, 2021 • edited Loading

znd4 commented Nov 22, 2021

morgans2 commented Dec 6, 2021

cpolizzi commented Dec 18, 2021

Configure Proxy Support Globally for All Systemd Services

Allow Interactive Login Sessions to VM to Use Proxy

Configure Trusted Certificates

Restart Managed Machine

kishorekkota commented Jan 3, 2022

rhatdan commented Jan 3, 2022

kishorekkota commented Jan 3, 2022

ctml91 commented Jan 3, 2022

cdnB8kn commented Jan 4, 2022

gclawes commented Jan 4, 2022 • edited Loading

matt314 commented Jan 4, 2022

mheon commented Jan 4, 2022

gclawes commented Jan 4, 2022

JdKock commented Feb 8, 2022 • edited Loading

gclawes commented Feb 8, 2022

matt314 commented Feb 11, 2022

mdavis-xyz commented Nov 3, 2022

mdavis-xyz commented Nov 3, 2022

cpolizzi commented Nov 3, 2022

cpolizzi commented Nov 3, 2022

mdavis-xyz commented Nov 4, 2022

gclawes commented Oct 12, 2021 •

edited

Loading

pboushy commented Oct 27, 2021 •

edited

Loading

pboushy commented Nov 5, 2021 •

edited

Loading

gclawes commented Jan 4, 2022 •

edited

Loading

JdKock commented Feb 8, 2022 •

edited

Loading