Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker run --rm --network leaves behind iptables rules when stopped #11324

Closed
vlk-charles opened this issue Aug 25, 2021 · 4 comments
Closed

docker run --rm --network leaves behind iptables rules when stopped #11324

vlk-charles opened this issue Aug 25, 2021 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@vlk-charles
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description
When docker run --rm is used to start a container in a non-default network (using --network) that is then stopped (and automatically removed), iptables rules created specifically for this container are left behind. This is especially bad when ports are published/forwarded (not demonstrated below), which causes DNAT rules to be added. When these are left behind, the host port is then unusable for future containers because the original rule takes precedence.

Steps to reproduce the issue:

# docker network create test
/etc/cni/net.d/test.conflist
# docker network ls
NETWORK ID    NAME        VERSION     PLUGINS
2f259bab93aa  podman      0.4.0       bridge,portmap,firewall,tuning
9f86d081884c  test        0.4.0       bridge,portmap,firewall,tuning,dnsname
# iptables -t nat -S | tee clean.iptables
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N CNI-HOSTPORT-DNAT
-N CNI-HOSTPORT-MASQ
-N CNI-HOSTPORT-SETMARK
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
# docker run --name nattest --network test -d busybox nc -l
360a0b8459d3280b9cc7a5e3dc05053baeb69ce564b1513a21453b55560164b3
# diff clean.iptables <(iptables -t nat -S) | grep '^<\|>'
> -N CNI-4fb5356a23a08b203f84e3a1
> -A POSTROUTING -s 10.89.1.3/32 -m comment --comment "name: \"test\" id: \"360a0b8459d3280b9cc7a5e3dc05053baeb69ce564b1513a21453b55560164b3\"" -j CNI-4fb5356a23a08b203f84e3a1
> -A CNI-4fb5356a23a08b203f84e3a1 -d 10.89.1.0/24 -m comment --comment "name: \"test\" id: \"360a0b8459d3280b9cc7a5e3dc05053baeb69ce564b1513a21453b55560164b3\"" -j ACCEPT
> -A CNI-4fb5356a23a08b203f84e3a1 ! -d 224.0.0.0/4 -m comment --comment "name: \"test\" id: \"360a0b8459d3280b9cc7a5e3dc05053baeb69ce564b1513a21453b55560164b3\"" -j MASQUERADE
# docker stop nattest
nattest
# docker rm nattest
360a0b8459d3280b9cc7a5e3dc05053baeb69ce564b1513a21453b55560164b3
# diff clean.iptables <(iptables -t nat -S) | grep '^<\|>'
# docker run --rm --name nattest --network test -d busybox nc -l
097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429
# diff clean.iptables <(iptables -t nat -S) | grep '^<\|>'
> -N CNI-3b4afbc074bb2197d7afd469
> -A POSTROUTING -s 10.89.1.4/32 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j CNI-3b4afbc074bb2197d7afd469
> -A CNI-3b4afbc074bb2197d7afd469 -d 10.89.1.0/24 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j ACCEPT
> -A CNI-3b4afbc074bb2197d7afd469 ! -d 224.0.0.0/4 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j MASQUERADE
# docker stop nattest
nattest
# docker ps -a
CONTAINER ID  IMAGE                           COMMAND               CREATED      STATUS          PORTS                   NAMES
# diff clean.iptables <(iptables -t nat -S) | grep '^<\|>'
> -N CNI-3b4afbc074bb2197d7afd469
> -A POSTROUTING -s 10.89.1.4/32 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j CNI-3b4afbc074bb2197d7afd469
> -A CNI-3b4afbc074bb2197d7afd469 -d 10.89.1.0/24 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j ACCEPT
> -A CNI-3b4afbc074bb2197d7afd469 ! -d 224.0.0.0/4 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j MASQUERADE
# docker run --rm --name nattest --network test -d busybox nc -l
6445cd6a359a8e14f1d42ee0ef518e152c4924599f484982016f11da599524cc
# diff clean.iptables <(iptables -t nat -S) | grep '^<\|>'
> -N CNI-3b4afbc074bb2197d7afd469
> -N CNI-dbd94516996e342139d942f4
> -A POSTROUTING -s 10.89.1.4/32 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j CNI-3b4afbc074bb2197d7afd469
> -A POSTROUTING -s 10.89.1.5/32 -m comment --comment "name: \"test\" id: \"6445cd6a359a8e14f1d42ee0ef518e152c4924599f484982016f11da599524cc\"" -j CNI-dbd94516996e342139d942f4
> -A CNI-3b4afbc074bb2197d7afd469 -d 10.89.1.0/24 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j ACCEPT
> -A CNI-3b4afbc074bb2197d7afd469 ! -d 224.0.0.0/4 -m comment --comment "name: \"test\" id: \"097e2099e769ac5b3dfe99aa7bee345773dfb087354e210a5c518358ca4c1429\"" -j MASQUERADE
> -A CNI-dbd94516996e342139d942f4 -d 10.89.1.0/24 -m comment --comment "name: \"test\" id: \"6445cd6a359a8e14f1d42ee0ef518e152c4924599f484982016f11da599524cc\"" -j ACCEPT
> -A CNI-dbd94516996e342139d942f4 ! -d 224.0.0.0/4 -m comment --comment "name: \"test\" id: \"6445cd6a359a8e14f1d42ee0ef518e152c4924599f484982016f11da599524cc\"" -j MASQUERADE

Describe the results you received:
Rules related to CNI-3b4afbc074bb2197d7afd469 remain after the original container has been stopped and removed, as seen in the last two invocations of iptables -t nat -S.

Describe the results you expected:
The rules should only exist for the lifetime of the container. The NAT table should return to the original state after the container is stopped.

Additional information you deem important (e.g. issue happens only occasionally):
Does not happen without --rm or --network, or at least not consistently.

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.16.6
Built:        Mon Aug  2 19:39:21 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.6-200.fc34.x86_64
  linkmode: dynamic
  memFree: 172097536
  memTotal: 2061930496
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 257h 12m 45.01s (Approximately 10.71 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 27
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627933161
  BuiltTime: Mon Aug  2 19:39:21 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman{,-plugins,-docker} container{s-common,networking-plugins} crun
podman-3.2.3-2.fc34.x86_64
podman-plugins-3.2.3-2.fc34.x86_64
podman-docker-3.2.3-2.fc34.noarch
containers-common-1-21.fc34.noarch
containernetworking-plugins-1.0.0-0.3.rc1.fc34.x86_64
crun-0.21-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes. Latest Fedora release with latest package from its repository.

Additional environment details (AWS, VirtualBox, physical, etc.):

Virtual machine on VMware ESXi. OS image from https://download.fedoraproject.org/pub/fedora/linux/releases/34/Cloud/x86_64/images/Fedora-Cloud-Base-34-1.2.x86_64.raw.xz .

# cat /etc/os-release 
NAME=Fedora
VERSION="34 (Cloud Edition)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34 (Cloud Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Cloud Edition"
VARIANT_ID=cloud
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 25, 2021
@Luap99
Copy link
Member

Luap99 commented Aug 26, 2021

This should be fixed with podman 3.3.
As workaround you have to use unset XDG_RUNTIME_DIR as root before running podman.

@Luap99 Luap99 closed this as completed Aug 26, 2021
@vlk-charles
Copy link
Author

Podman 3.3 made it into Fedora 34 about one day after my report.[1] I updated, tested and can confirm that the bug is not present anymore. Thank you.

@vlk-charles
Copy link
Author

Nevermind. It still happens under some circumstances, just not the ones described in the original post. I will try to find a set of steps for consistent reproduction.

@vlk-charles vlk-charles mentioned this issue Sep 1, 2021
Closed
@vlk-charles
Copy link
Author

Just for full disclosure, this bug is fixed. My continued issues have a different cause and #11384 was opened for it.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vlk-charles @Luap99