[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container #10929

AkihiroSuda · 2021-07-14T09:32:38Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container

Regression in v3.2.0.

Steps to reproduce the issue:

Terminal 1: Run a rootless container with CNI-in-slirp4netns

$ podman network create foo
$ podman run -it --rm --net=foo alpine
/ # wget -O- example.com
Connecting to example.com (93.184.216.34:80)
writing to stdout
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...
</html>
-                    100% |*****************************************************************************************************************************************************************************************************|  1256  0:00:00 ETA
written to stdout

Terminal 2: Run some rootful container

$ sudo podman run --rm hello-world

Terminal 1: DNS gets broken in the rootless container

/ # wget -O- example.com
wget: bad address 'example.com'

Describe the results you received:

wget: bad address 'example.com'

Describe the results you expected:

Should work

Additional information you deem important (e.g. issue happens only occasionally):

v3.1.2 works as expected. v3.2.0 and main (9d98f56 ) is broken.
Non-CNI rootless container works as expected

Output of podman version:

Version:      3.3.0-dev
API Version:  3.3.0-dev
Go Version:   go1.16.6
Git Commit:   9d98f56a64852d3bf9d3b27ad6e1ac3e3ebb6622
Built:        Wed Jul 14 18:29:59 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.0-dev
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 2.0.26, commit: 0e155c83aa739ef0a0540ec9f9d265f57f68038b'
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "21.04"
  eventLogger: journald
  hostname: suda-ws01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.0-22-generic
  linkmode: dynamic
  memFree: 5607809024
  memTotal: 16748163072
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 0.19-dirty
      commit: e67a75672412975916dac6b87f8346f870e4b99a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/local/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.11
      commit: 368e69ccc074628d17a9bb9a35b8f4b9f74db4c6
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 10h 15m 42.1s (Approximately 0.42 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/suda/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/local/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/suda/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 109
  runRoot: /tmp/1001
  volumePath: /home/suda/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0-dev
  Built: 1626254999
  BuiltTime: Wed Jul 14 18:29:59 2021
  GitCommit: 9d98f56a64852d3bf9d3b27ad6e1ac3e3ebb6622
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.0-dev

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

The text was updated successfully, but these errors were encountered:

AkihiroSuda · 2021-07-14T09:43:30Z

Before running a rootful container:

$ cat /proc/$(pgrep -u $(id -u) dnsmasq)/root/etc/resolv.conf
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
options edns0 trust-ad

After running a rootful container:

$ cat /proc/$(pgrep -u $(id -u) dnsmasq)/root/etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

The bind-mount for /etc/resolv.conf is broken because systemd-resolved recreates /etc/resolv.conf on the host when a network stuff has changed in the host network namespace.

Rootless Podman v3.1.2 and Rootless Docker can work as expected because they don't bind-mount /etc/resolv/conf from the host.

Luap99 · 2021-07-14T11:22:28Z

I cannot reproduce.
Rootless CNI creates a resolv.conf at $XDG_RUNTIME_DIR/rootless-cni/resolv.conf. This will be bind mounted to /etc/resolv.conf.

I think your example is incorrect because /etc/resolv.conf is a symlink. Not sure why it would work before the root container.

ls -l /proc/$(pgrep -u $(id -u) dnsmasq)/root/etc/resolv.conf
lrwxrwxrwx. 1 root root 37 22. Mär 13:25 /proc/45061/root/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
cat /proc/$(pgrep -u $(id -u) dnsmasq)/root/run/systemd/resolve/stub-resolv.conf
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
options edns0 trust-ad

You can join the correct namespace with podman unshare --rootless-cni.

AkihiroSuda · 2021-07-15T06:50:00Z

I think your example is incorrect because /etc/resolv.conf is a symlink. Not sure why it would work before the root container.

$ ls -l /proc/$(pgrep -u $(id -u) dnsmasq)/root/etc/resolv.conf
lrwxrwxrwx 1 root root 39 May  5  2019 /proc/5358/root/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

The link points to ../run/systemd/resolve/stub-resolv.conf before and after.

Before running a rootful container, /run/systemd/resolve/stub-resolv.conf is a bind-mounted file with nameserver 10.0.2.3, 8.8.8.8, and 8.8.4.4.
After running a rootful container, the file is unmounted (because the mount point /run/systemd/resolve/stub-resolv.conf disappears and then gets recreated), so /run/systemd/resolve/stub-resolv.conf on the host with 127.0.0.53 appears.

You can join the correct namespace with podman unshare --rootless-cni.

This seems to create a new mount namespace, so I don't think this command can be used for testing.

AkihiroSuda · 2021-07-15T07:46:56Z

I cannot reproduce.

Does this reproduce (or any command that results in removal and recreation of /run/systemd/resolve/stub-resolv.conf)?

$ sudo mv /run/systemd/resolve/stub-resolv.conf /tmp/
$ sudo mv /tmp/stub-resolv.conf /run/systemd/resolve/

Luap99 · 2021-07-15T08:02:24Z

Does this reproduce (or any command that results in removal and recreation of /run/systemd/resolve/stub-resolv.conf)?
$ sudo mv /run/systemd/resolve/stub-resolv.conf /tmp/
$ sudo mv /tmp/stub-resolv.conf /run/systemd/resolve/

Yes this works.

You can join the correct namespace with podman unshare --rootless-cni.

This seems to create a new mount namespace, so I don't think this command can be used for testing.

You are right, podman unshare nsenter -m -t $(pgrep -u $(id -u) dnsmasq) should work to join the mount ns.

OK, I understand the problem but how can podman fix this?

AkihiroSuda · 2021-07-15T08:14:41Z

PR: #10936

AkihiroSuda added 3.2 CNI Bug with CNI networking for root containers kind/bug Categorizes issue or PR as related to a bug. rootless labels Jul 14, 2021

AkihiroSuda mentioned this issue Jul 15, 2021

CNI-in-slirp4netns: fix bind-mount for /run/systemd/resolve/stub-resolv.conf #10936

Merged

openshift-merge-robot closed this as completed in #10936 Jul 15, 2021

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container #10929

[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container #10929

AkihiroSuda commented Jul 14, 2021 •

edited

Loading

AkihiroSuda commented Jul 14, 2021

Luap99 commented Jul 14, 2021

AkihiroSuda commented Jul 15, 2021 •

edited

Loading

AkihiroSuda commented Jul 15, 2021

Luap99 commented Jul 15, 2021

AkihiroSuda commented Jul 15, 2021

[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container #10929

[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container #10929

Comments

AkihiroSuda commented Jul 14, 2021 • edited Loading

AkihiroSuda commented Jul 14, 2021

Luap99 commented Jul 14, 2021

AkihiroSuda commented Jul 15, 2021 • edited Loading

AkihiroSuda commented Jul 15, 2021

Luap99 commented Jul 15, 2021

AkihiroSuda commented Jul 15, 2021

AkihiroSuda commented Jul 14, 2021 •

edited

Loading

AkihiroSuda commented Jul 15, 2021 •

edited

Loading