DNS and other errors after original network is disconnected

[linggao]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Start a container with a network, connect it to another network and disconnect it from the original network it started with. The DNS is lost for the container and it cannot even ping with an ip.

Steps to reproduce the issue:

1. podman network create foo-a
   podman network create foo-b

2. podman run --name test --network foo-a -d alpine sleep 10000

3. podman network connect foo-b test
   podman network disconnect foo-a test

4. podman exec -it test ping google.com
   ping: bad address 'google.com'.

5. podman exec -it test ping 172.217.0.238
   PING 172.217.0.238 (172.217.0.238): 56 data bytes
   ping: sendto: Network unreachable
   WARN[0000] Error resizing exec session 79bb31f01304ce475be90f0eabaf55fbb40c1fafd7bf8822419a5d3b19f3eb33: could not open ctl file for terminal resize for container a5092f2db3f45dde2e785fae5ef2ba4c70843a5e0c220744fe1f64a4e4234503: open /var/lib/containers/storage/overlay-containers/a5092f2db3f45dde2e785fae5ef2ba4c70843a5e0c220744fe1f64a4e4234503/userdata/79bb31f01304ce475be90f0eabaf55fbb40c1fafd7bf8822419a5d3b19f3eb33/ctl: no such device or address

Describe the results you received:
Please see output from step 4 and 5.

Describe the results you expected:
No errors.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.0-dev
API Version:  3.0.0
Go Version:   go1.14.12
Git Commit:   426178a49991106ffe222f12cc42409ae78dd257-dirty
Built:        Tue Mar  2 16:08:11 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.27-dev, commit: 7310bf13319ee8ed50799b202509bedc27b36cf8'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: lingvs4.dev.edge-fabric.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-240.15.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 2636328960
  memTotal: 8342470656
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93+dev
      commit: f245a1d1edbf545549e5a16106cf1aec356a3c7d
      spec: 1.0.2-dev
      go: go1.14.12
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2146758656
  swapTotal: 2146758656
  uptime: 316h 24m 18.01s (Approximately 13.17 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 10
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614722891
  BuiltTime: Tue Mar  2 16:08:11 2021
  GitCommit: 426178a49991106ffe222f12cc42409ae78dd257-dirty
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.1.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman is built from the latest master.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[baude]

what is 172.217.0.238 ?

[baude]

Ok, here is the diagnosis. There are two bugs here.

When I reproduce this, I think the dns is somewhat expected? but maybe actually a bug. When connecting to the first network, you get a /etc/resolv.conf pointing to the gateway of the first network. when you connect to the second network, that is unchanged. when we remove the first network, /etc/resolv.conf is unchanged.

There seems to be a secondary problem where when adding a second network, no route is defined. So when the first network is removed, the container seems to loose it's default route.

[baude]

additionally, if you disconnect the first network and then connect the second, a default route is defined. (and again, the resolve is unchanged).

@mheon lets talk about this one ...

[linggao]

@baude 172.217.0.238 is the IP address returned when I ping google.com.

[jarcher]

What are all of the flags you are using for managing the containers with the hzn client? are yall using the network-alias, link and/or dns flags from the docker version?

[linggao]

@jarcher we are using go-dockerclient to talk to the API endpoint. We definitely use network alias. Not dns or link. Here is an example of our container using docker inspect.

    {
        "Id": "f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad",
        "Created": "2021-03-02T17:38:11.066186717Z",
        "Path": "ping",
        "Args": [
            "google.com"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2264433,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-03-02T17:38:12.074663553Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:e651fe6a248e9f96352e83878fdf943969e3eea2032b3b135458a1988161ff2c",
        "ResolvConfPath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/hostname",
        "HostsPath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/hosts",
        "LogPath": "",
        "Name": "/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d-ling_podman_dep1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/tmp/testdata:/tmp/mydata:ro",
                "myvolume1:/tmp/mydata2",
                "/tmp/hzndev:/tmp/hzndev",
                "/tmp/hzndev/auth/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d:/ess-auth:ro",
                "/tmp/hzndev/auth/SSL/cert:/ess-cert:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "tag": "workload-anaxsquad_lingpodmandep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_ling_podman_dep1"
                }
            },
            "NetworkMode": "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
            "PortBindings": {
                "7777/udp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": ""
                    }
                ],
                "8888/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "SYS_ADMIN",
                "SYS_TIME"
            ],
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": [
                "1007"
            ],
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "Tmpfs": {
                "/app": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 4294967296,
            "NanoCpus": 1500000000,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/bus/usb/001/001",
                    "PathInContainer": "/dev/bus/usb/001/003",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 8589934592,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f-init/diff:/var/lib/docker/overlay2/38a1167ffa9609029aaf061988641c803276d536c479c18879ca5533e43954a8/diff:/var/lib/docker/overlay2/394a8b350f89a16ebf4fee478941335cd28337d20d51b7723fd9c24367eabd71/diff",
                "MergedDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/merged",
                "UpperDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/diff",
                "WorkDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/tmp/hzndev",
                "Destination": "/tmp/hzndev",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/hzndev/auth/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
                "Destination": "/ess-auth",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/hzndev/auth/SSL/cert",
                "Destination": "/ess-cert",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/testdata",
                "Destination": "/tmp/mydata",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "myvolume1",
                "Source": "/var/lib/docker/volumes/myvolume1/_data",
                "Destination": "/tmp/mydata2",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "f7c60861c01a",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "7777/udp": {},
                "8888/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "HZN_ESS_API_ADDRESS=/tmp/hzndev/essapi.sock",
                "HZN_ESS_AUTH=/ess-auth/auth.json",
                "HZN_ESS_CERT=/ess-cert/cert.pem",
                "HZN_PATTERN=",
                "HZN_ARCH=amd64",
                "HZN_PRIVILEGED=false",
                "HW_WHO=World Dep1",
                "HZN_DEVICE_ID=lingvs2",
                "HZN_ESS_API_PROTOCOL=secure-unix",
                "HZN_ESS_API_PORT=0",
                "HZN_CPUS=4",
                "HZN_HOST_IPS=127.0.0.1,10.167.11.74,169.55.186.244,172.17.0.1",
                "HZN_EXCHANGE_URL=https://cp-console.ieam-roks-stg-70ea81cdef68a2eb78ece6d890b7dad3-0000.us-south.containers.appdomain.cloud/edge-exchange/v1",
                "HZN_RAM=32156",
                "HZN_ORGANIZATION=anaxsquad",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "google.com"
            ],
            "Image": "linggao/ling_podman_dep1_amd64:0.0.1",
            "Volumes": {
                "/ess-auth": {},
                "/ess-cert": {},
                "/tmp/hzndev": {},
                "/tmp/mydata": {},
                "/tmp/mydata2": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "ping"
            ],
            "OnBuild": null,
            "Labels": {
                "openhorizon.anax.agreement_id": "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
                "openhorizon.anax.deployment_description_hash": "d-2JFGJPZWhXCQhFqwyE-kam1zU=",
                "openhorizon.anax.dev_service": "true",
                "openhorizon.anax.infrastructure": "",
                "openhorizon.anax.service_name": "ling_podman_dep1",
                "openhorizon.anax.variation": ""
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "aabb3c5a05efe168c36dc93c5cd102db6f0fb5d335835f27ebcaff87db7c842a",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "7777/udp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "32917"
                    }
                ],
                "8888/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32931"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/aabb3c5a05ef",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_2827da80bb7a652e540135cf416823ebfc5c94df16ee2121896a097be4d94f93": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "ling_podman_dep1",
                        "f7c60861c01a"
                    ],
                    "NetworkID": "8b1c5c67719541c282c4fa5e73882040e229dcf3dc7047db2e84bc712443fd5f",
                    "EndpointID": "e0413f7bb9b797c7b48b7966f03be664c5ac507007247b00e9634e3f71e8f272",
                    "Gateway": "172.28.0.1",
                    "IPAddress": "172.28.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:1c:00:02",
                    "DriverOpts": null
                },
                "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_anaxsquad_lingPodmanDep2_0.0.1_b3bad5e9-48a0-436d-90f1-4fe8fe708cb5": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "ling_podman_dep1",
                        "f7c60861c01a"
                    ],
                    "NetworkID": "83b8688c7293361058d2bdcf7e57e5228776eefd7ecdaeb0e146b2831a5d3d39",
                    "EndpointID": "d8b8ffc8ef52051b37eb4283119cef95fd327c199009da2c2f56d5ea8681f3e4",
                    "Gateway": "172.29.0.1",
                    "IPAddress": "172.29.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:1d:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

[mccv1r0]

Issue opened in cni plugins; containernetworking/plugins#601

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[umohnani8]

the issue in cni plugins was fixed and closed, is this still an issue here @baude @mheon @linggao?

[mheon]

Unsure. It is definitely a target of our current network rewrite efforts, though.

[linggao]

@umohnani8 I am okay to close it if the scenario described in the issue is working. Namely:

podman network create foo-a
podman network create foo-b

podman run --name test --network foo-a -d alpine sleep 10000

podman network connect foo-b test
podman network disconnect foo-a test

podman exec -it test ping google.com

I do not have the env to test it now.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

this should be fixed with aardvark and podman 4.0 release

[baude]

works with podman 4

[baude]

i want to double check this one