Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rootful containers when using corporate VPN on systemd-resolved systems have broken DNS #17075

Closed
jiridanek opened this issue Jan 11, 2023 · 12 comments
Closed

Rootful containers when using corporate VPN on systemd-resolved systems have broken DNS #17075

jiridanek opened this issue Jan 11, 2023 · 12 comments
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@jiridanek
Copy link 

    A `/etc/resolv.conf` generated by systemd-resolved looks like this:

nameserver 127.0.0.53
options edns0
search some.dom

As a result podman seems to remove the nameserver line and adds the "upstream" DNS servers directly.

That is, it adds content of /run/systemd/resolve/resolv.conf, which may look like this

nameserver 192.168.86.1
search some.dom

Bypassing systemd-resolved on the host may work in some scenarios, but it breaks others.
Consider a corporate VPN connections where "upstream" is not defined. (We have at least two "upstreams" when connected)

Bypassing the host's systemd-resolved has at least the following problems:

  • Upstream/Internet DNS servers do not know about corporate specific DNS entries
  • Corporate DNS servers may deliver different addresses than public DNS servers
  • Corporate DNS servers may not deliver results for public stuff
  • Corporate DNS servers may change, so unable to use fixed IP addresses

So whatever which DNS server I choose by setting --dns=<ip>, I will never get the same results as talking to the systemd-resolved running on the host.

How to fix that?

Well, I guess it's not possible at all. podman would have to replace nameserver 127.0.0.53 with something that is forwarded or hosted on the host. But systemd-resolved is listening on the loopback interface only and does not allow (AFAIK) to change / configure that.

The combination of systemd-resolved and podman is the default for Fedora users.
So privileged containers are quite unusable when corporate VPNs are in the game.

Originally posted by @fbezdeka in #3277 (comment)
@jiridanek jiridanek mentioned this issue Jan 11, 2023
Closed
@jiridanek
Copy link
Author

Reproducer, using Red Hat corporate VPN on my laptop

$ sudo podman run --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
curl: (6) Could not resolve host: download-node-02.eng.bos.redhat.com
$ podman run --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
...

That is, if I run podman under sudo, I cannot reach server on corporate VPN. But if I run it as rootless podman, then some trickery with aardvark-dns and the 10.0.2.3 dns server allows it to work.

@Luap99
Copy link
Member

Luap99 commented Jan 11, 2023

You can enable dns on your network (or create a network with podman network create) which will make it use aardvark-dns (assuming netavark network backend), this will act as proxy and is able to reach 127.0.0.1.

@jiridanek
Copy link
Author

@Luap99 Thanks, that works!

$ sudo podman network create pepa
$ sudo podman run --net=pepa --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
...

Is there any truth in #3277 (comment), suggesting that in Podman 4.4 this is intended to work without custom network? If yes, then I think this issue can track that (unless there is another issue for it). Otherwise, I'm happy with creating a named net for my rootful containers and consider this solved.

@Luap99
Copy link
Member

Luap99 commented Jan 11, 2023

Without having dns enabled it will not work, we need some form of proxy to reach localhost on the host. I would say this is the expected behaviour right now. The other way is to use --network host but this has more downsides of course.

@jiridanek
Copy link
Author

jiridanek commented Jan 11, 2023 •
edited
Loading

edit: yeah, I can't use --net=host, because I rely on my containers having an IP that is reachable from the host machine, that is the whole reason I use rootful containers in the first place. I am using containers to fake a VM for running my tests.

Another thing that works is I can list the DNS servers explicitly, but beware, on my system the order matters, if I list VPN dns servers first, all works, but if I list my home network DNS first, then resolving on that fails the resolution without moving to VPN dns servers...

podman run --dns=10.xxx.xxx.xxx --dns=10.xxx.xxx.xxx --dns=192.168.xxx.xxx [...]

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@jiridanek
Copy link
Author

@mheon Is there any truth in your #3277 (comment), suggesting that in Podman 4.4 this is intended to work without custom network? If yes, then I think this issue can track that (unless there is another issue for it). Otherwise, I'm happy with creating a named net for my rootful containers and consider this solved.

@mheon
Copy link
Member

mheon commented Feb 12, 2023

It requires a custom network - 4.4 was required to allow this to work at all.

@jiridanek
Copy link
Author

@mheon I have podman version 4.3.1 and I got it to work with the custom network, using the commands in #17075 (comment).

@rhatdan
Copy link
Member

rhatdan commented Feb 13, 2023

Can we close this issue now?

@jiridanek
Copy link
Author

I'd like to understand the podman 4.4 comment better, and whether there are additional improvements planned (for my original problem: without defining custom podman network, rootful containers "don't work", while rootless containers "work"). But over all, my problem has a solution (define custom network when using rootful podman), so I would not mind calling it quits here.

@Luap99
Copy link
Member

Luap99 commented Feb 13, 2023

Yes you need a custom network with dns enabled, there is no other way at the moment.
You could set the new network as default in containers.conf as described here: https://blog.podman.io/2023/02/the-container-name-resolution-conundrum/

@Luap99 Luap99 closed this as completed Feb 13, 2023
@energister energister mentioned this issue Jun 21, 2023
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 1, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jiridanek @rhatdan @mheon @Luap99