Duplicated settings in /etc/resolv.conf when set via containers.conf (Ubuntu 20.04)

[josi19]

/kind bug

Description

There seems to be a bug with custom DNS settings set through containers.conf files in Ubuntu 20.04 (and maybe higher).
Settings, which are applied via containers.conf files are being duplicated in /etc/resolv.conf.

Steps to reproduce the issue:

The testing servers:

root@josi-test-focal01:~ # lsb_release -d
Description:    Ubuntu 20.04.3 LTS

root@josi-test-bionic01:~ # lsb_release -d
Description:    Ubuntu 18.04.6 LTS

When the following configuration is set, all entries in the /etc/resolv.conf inside the freshly spawned containers are duplicated:

root@josi-test-focal01:~ # cat /etc/containers/containers.conf
[containers]
  dns_searches  = [ "example.com"]
  dns_servers = [
    "1.1.1.1",
    "1.0.0.1",
    "8.8.8.8",
]

root@josi-test-focal01:~ # podman run --rm -it ubuntu
root@729a256414a1:/# cat /etc/resolv.conf
search example.com example.com
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8

If the same is done on the system with Ubuntu 18.04, it works as expected:

root@josi-test-bionic01:~ # podman run --rm -it ubuntu
root@f8c36ec0912a:/# cat /etc/resolv.conf
search example.com
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8

However, when I additionally set a DNS server via the command line, the nameservers are no longer duplicated but the dns_search parameter still is:

root@josi-test-focal01:~ # podman run --rm --dns=1.2.3.4 -it ubuntu
root@6317296d4977:/# cat /etc/resolv.conf
search example.com example.com
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 1.2.3.4

If I add the flag --dns-search to the command too, I get these results:

root@josi-test-focal01:~ # podman run --rm --dns=1.2.3.4 --dns-search=example.com -it ubuntu
root@e71c479a9eda:/# cat /etc/resolv.conf
search example.com example.com
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 1.2.3.4

root@josi-test-focal01:~ # podman run --rm --dns=1.2.3.4 --dns-search=mydomain.org -it ubuntu
root@d950cd05843e:/# cat /etc/resolv.conf
search example.com mydomain.org
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 1.2.3.4

There are no additional settings regarding DNS either in $HOME/.config/containers/containers.conf nor in /usr/share/containers/containers.conf.

Describe the results you received:

already described above

Describe the results you expected:

The behavior in Ubuntu 18.04 is correct and it should be the same result in newer Ubuntu versions.

Additional information you deem important (e.g. issue happens only occasionally):

Can be reproduced every time.
Since one of the bigger differences between our Bionic and Focal instances is the storage engine, I tried it with vfs and overlay but the behavior was the same.

Output of podman version:

root@josi-test-focal01:~ # podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

root@josi-test-bionic01:~ # podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

root@josi-test-focal01:~ # podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 6
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: josi-test-focal01
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-96-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 4428910592
  memTotal: 8300314624
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 3221221376
  swapTotal: 3221221376
  uptime: 51h 40m 9.11s (Approximately 2.12 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /root
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /root
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

root@josi-test-bionic01:~ # podman info --debug
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname: josi-test-bionic01
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.15.0-161-generic
  linkmode: dynamic
  memFree: 298356736
  memTotal: 3144806400
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2852646912
  swapTotal: 3070226432
  uptime: 1945h 17m 35.57s (Approximately 81.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 1
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

root@josi-test-focal01:~ # apt list podman
Listing... Done
podman/unknown,now 100:3.4.2-1 amd64 [installed]
podman/unknown 100:3.4.2-1 arm64
podman/unknown 100:3.4.2-1 armhf
podman/unknown 100:3.4.2-1 s390x

root@josi-test-bionic01:~ # apt list podman
Listing... Done
podman/unknown,now 100:3.0.1-2 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes --> No mentioning of this issue in the troubleshooting guide and tested with the newest version available via the official repository. However, not tested with the newest version 3.4.4 resp. 4.0, since these are not yet available through a repository.

Additional environment details (AWS, VirtualBox, physical, etc.):

Virtual Machine (Nutanix with AHV supervisor)

[josi19]

I now manually compiled Podman with the version 4.0.0-dev and the problem seems to exist in this version too:

root@josi-test-focal01:~ # podman version
Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.6
Git Commit:   1b544b74247e538a2cda7bd476cb340cf8f57b81
Built:        Fri Jan 28 15:30:23 2022
OS/Arch:      linux/amd64

root@josi-test-focal01:~ # podman run --rm -it ubuntu
root@570f847014cf:/# cat /etc/resolv.conf
search example.com example.com
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8

[josi19]

It seems to work as expected with the older version 3.2.3 on Ubuntu Focal.
The problem was probably introduced with the merge of this PR: #11360