Rootful podman cni dns resolution defaults to 8.8.8.8, are there ways for it to auto detect the ones the host is using?

[KCSesh]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Disclaimer: I don't understand all of the network details of the host I am accessing in an offsite location.

I am accessing a host through a VPN which has podman.
The host seems to have a local nameserver running:

Host file:

cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.53

If I were to change this file to use the nameserver 8.8.8.8, the host will not properly do DNS. And simple commands like: curl google.com stop working.

So when I start a Podman container, it does not have DNS because the /etc/resolv.conf looks like this by default:

[root@fc20b160c79a /]# cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

If I do the same test with Docker, docker seems to be able to fetch the DNS hosts that 127.0.0.53, and docker containers run with DNS working on the exact same host with no changes.

This is the docker: /etc/resolve.conf
Full IP's are omitted.

sudo docker run --rm  -it fedora
[root@b7809e842f47 /]# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.---.--.--
nameserver 10.---.--.--
# Too many DNS servers configured, the following entries may be ignored.
nameserver 10.---.---.--
nameserver 10.---.--.--
search nvidia.com 

I I manually set the DNS of my podman container with the runtime flag of --dns=10.---.--.-- and DNS is fixed in the podman container.

I would like to know if there is a plan to make the CNI detect the actual DNS instead of what seems to be a hardcoding of 8.8.8.8

Steps to reproduce the issue:

1. Have a host that uses a local nameserver like nameserver 127.0.0.53 in /etc/resolv.conf

2.sudo podman run -it fedora /bin/bash

3. curl google.com fails.

Describe the results you received:
Podman container start without DNS.
I can curl IP's directly, but curls like:
curl google.com and curl example.com fail.

Describe the results you expected:
I would expect podman CNI to detect my actual hosts DNS servers and have my container work as expected by default, just like Docker.

** Additional Details **

On the host if I run the command:

systemd-resolve --status

I see the output:

Link 3 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.---.--.--
                      10.---.--.--
                      10.---.---.--
                      10.---.--.---
          DNS Domain: ~.
                      **webstie**.com

These Ip's match those found in Docker.
Could podman grab this data?

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.16
Git Commit:   2c9d72766b051455aa0a1b8f9be1280207220859-dirty
Built:        Fri May 28 17:22:58 2021
OS/Arch:      linux/arm64

Output of podman info --debug:

host:
  arch: arm64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/local/bin/conmon'
    path: /usr/local/bin/conmon
    version: 'conmon version 2.0.29-dev, commit: '
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname:ubuntu
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.9.140
  linkmode: dynamic
  memFree: 15453519872
  memTotal: 33477275648
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.13.8
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16738615296
  swapTotal: 16738615296
  uptime: 26h 2m 31.57s (Approximately 1.08 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 21
    paused: 0
    running: 0
    stopped: 21
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /data/podman/podman-root
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /data/podman/podman-run-root
  volumePath: /data/podman/podman-root/volumes
version:
  APIVersion: 3.0.0
  Built: 1622247778
  BuiltTime: Fri May 28 17:22:58 2021
  GitCommit: 2c9d72766b051455aa0a1b8f9be1280207220859-dirty
  GoVersion: go1.16
  OsArch: linux/arm64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

Build from self created .deb for ARM on ubuntu 18.04

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No only 3.0.1

Additional environment details (AWS, VirtualBox, physical, etc.):
Host behind VPN and special Network.

[Luap99]

Podman should already do this. All nameservers in the ´127.0.0.0/8´ range are filtered, if there are no nameservers left 8.8.8.8 is used.
I noticed that you are using systemd-resolved. Podman should also be able to detect this as long as /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf.

[mheon]

Knowing what /etc/resolv.conf is (is it a symlink, if so what does it point to) would help here. I suspect that Ubuntu may be doing a different systemd-resolved configuration than Fedora and it's throwing off our detection of that.

[intelfx]

Judging by contents of resolv.conf pasted earlier, it is probably a symlink to /run/systemd/resolve/resolv.conf (in resolved, that file contains all upstream servers). But, according to the same paste, it only contains nameservers in 10.0.0.0/8, so they shouldn't be filtered then?

[KCSesh]

I noticed that you are using systemd-resolved. Podman should also be able to detect this as long as /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf.

The file is not symlinked.

~$ sudo ls -la /etc/resolv.conf
-rw-r--r-- 1 root root 105 Jun  4 18:29 /etc/resolv.conf

However, if I manually run:

sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Podman DNS is fixed.

But this still is showing Docker is doing something unique, right?

cat /run/systemd/resolve/stub-resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search nvidia.com <internal-reference>.com

Just for clarity, with docker the file does not need to be symlinked.
With podman, symlink the file does fix it.

I can investigate if my host should have this symlinked, and if that is a miss happening on the host-image.

[mheon]

Hm. So systemd-resolvd is in use, but system resolv.conf is not a symlink? That's new.

I wonder how we can detect this case.

[KCSesh]

Update: There seems to be a lot of discussion around these files in the ubuntu world. I am not sure if the links are exactly related, however, it does seem that by default that symlink does not exist.
And it seems often a fix is to create the symlink manually.
DNS works on the host without the symlink, so it should not be needed.

Original:
From: https://askubuntu.com/questions/973017/wrong-nameserver-set-by-resolvconf-and-networkmanager
It seems to be a bug: https://bugs.launchpad.net/ubuntu/+source/network-manager-strongswan/+bug/1864256
And the proposed fix, is to create the symlink yourself: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320/comments/8

I'll have to investigate this a bit more, but it seems that my host should be symlinked.
Somehow Docker is still detecting this?

[Luap99]

Docker just checks if 127.0.0.53 is the only nameserver in /etc/resolv.conf, see https://github.com/moby/libnetwork/blob/b3507428be5b458cb0e2b4086b13531fb0706e46/resolvconf/resolvconf.go#L47-L51.

[mheon]

OK. That seems reasonable.