Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

portmap plugin is causing performance problems in Kubernetes due to iptables lock #510

Closed
aojea opened this issue Jul 13, 2020 · 0 comments · Fixed by #509
Closed

portmap plugin is causing performance problems in Kubernetes due to iptables lock #510

aojea opened this issue Jul 13, 2020 · 0 comments · Fixed by #509

Comments

@aojea
Copy link
Contributor

aojea commented Jul 13, 2020 •
edited
Loading

In a Kubernetes environment, using containerd, when a pod is deleted the portmap plugin is executed at least 3 times, despite it does not have portmaps configured. This has some consequences, because portmap executes iptables and ip6tables, that means that we hold the iptables lock at least 6 times.

how to repro

Create a Kubernetes deployment, i.e. kind create cluster

  1. move the portmap plugin to portmap.bin
  2. create a wrapper script to log the calls to the plugin
/opt/cni/bin/portmap
#!/bin/sh
echo $(date +"%T") >> /tmp/debug_portmap
echo "CNI: $CNI_COMMAND $CNI_IFNAME $CNI_NETNS $CNI_CONTAINERID"  >> /tmp/debug_portmap
tee -a /tmp/debug_portmap | /opt/cni/bin/portmap.bin
echo "\n---------------" >> /tmp/debug_portmap
  1. create a pod kubectl run nginx --image=nginx --port=80
CNI: ADD eth0 /var/run/netns/cni-5f0b2fce-72a5-6108-092f-241e4030e484 47aa93243add946867e7acb33291a483ea19f417bef324665c2f9669ede0b703
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","prevResult":{"cniVersion":"0.3.1","dns":{},"interfaces":[{"mac":"86:36:f5:fa:5b:9e","name":"vethf01dee95"},{"mac":"02:4c:5d:31:c8:a1","name":"eth0","sandbox":"/var/run/netns/cni-5f0b2fce-72a5-6108-092f-241e4030e484"}],"ips":[{"address":"10.244.0.21/24","gateway":"10.244.0.1","interface":1,"version":"4"}],"routes":[{"dst":"0.0.0.0/0"}]},"runtimeConfig":{"portMappings":null},"type":"portmap"}
---------------
  1. delete the pod kubectl delete deployment nginx
09:39:44
CNI: DEL eth0 /var/run/netns/cni-5f0b2fce-72a5-6108-092f-241e4030e484 47aa93243add946867e7acb33291a483ea19f417bef324665c2f9669ede0b703
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":null},"type":"portmap"}
---------------
09:39:45
CNI: DEL eth0  47aa93243add946867e7acb33291a483ea19f417bef324665c2f9669ede0b703
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":null},"type":"portmap"}
---------------
09:39:46
CNI: DEL eth0  47aa93243add946867e7acb33291a483ea19f417bef324665c2f9669ede0b703
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":null},"type":"portmap"}
---------------

So far, we see that our portmap plugin is called despite there is no hostPorts, let's create a pod with hostPorts

    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
          hostPort: 8080
09:39:43
CNI: ADD eth0 /var/run/netns/cni-0b43b484-5f8c-f5ee-45fc-799d285e8c52 587c678bc086c87e4e95748ac2e1a186d426d8ada6758450f7b8847a53bfc482
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","prevResult":{"cniVersion":"0.3.1","dns":{},"interfaces":[{"mac":"fe:ea:f7:1e:f5:7b","name":"vethecffc0b3"},{"mac":"72:69:36:b7:bb:1d","name":"eth0","sandbox":"/var/run/netns/cni-0b43b484-5f8c-f5ee-45fc-799d285e8c52"}],"ips":[{"address":"10.244.0.22/24","gateway":"10.244.0.1","interface":1,"version":"4"}],"routes":[{"dst":"0.0.0.0/0"}]},"runtimeConfig":{"portMappings":[{"HostPort":8080,"ContainerPort":80,"Protocol":"tcp","HostIP":""}]},"type":"portmap"}

we can observe that is executed one time more with a delay

09:42:41
CNI: DEL eth0 /var/run/netns/cni-0b43b484-5f8c-f5ee-45fc-799d285e8c52 587c678bc086c87e4e95748ac2e1a186d426d8ada6758450f7b8847a53bfc482
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":[{"HostPort":8080,"ContainerPort":80,"Protocol":"tcp","HostIP":""}]},"type":"portmap"}
---------------
09:42:42
CNI: DEL eth0  587c678bc086c87e4e95748ac2e1a186d426d8ada6758450f7b8847a53bfc482
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":[{"HostPort":8080,"ContainerPort":80,"Protocol":"tcp","HostIP":""}]},"type":"portmap"}
---------------
09:42:43
CNI: DEL eth0  587c678bc086c87e4e95748ac2e1a186d426d8ada6758450f7b8847a53bfc482
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":[{"HostPort":8080,"ContainerPort":80,"Protocol":"tcp","HostIP":""}]},"type":"portmap"}
---------------
09:43:31
CNI: DEL eth0  587c678bc086c87e4e95748ac2e1a186d426d8ada6758450f7b8847a53bfc482
{"capabilities":{"portMappings":true},"cniVersion":"0.3.1","name":"kindnet","runtimeConfig":{"portMappings":[{"HostPort":8080,"ContainerPort":80,"Protocol":"tcp","HostIP":""}]},"type":"portmap"}
---------------

so, it seems containerd tears down the network multiple times

Jul 13 09:13:15 kind-control-plane containerd[25550]: time="2020-07-13T09:13:15.937380950Z" level=info msg="TearDown network for sandbox \"b78efe047b461be029f0b2e8a08a1f478bfaee97a57c253945e5ca4289e38014\" successfully"
Jul 13 09:13:16 kind-control-plane containerd[25550]: time="2020-07-13T09:13:16.859755495Z" level=info msg="TearDown network for sandbox \"b78efe047b461be029f0b2e8a08a1f478bfaee97a57c253945e5ca4289e38014\" successfully"
Jul 13 09:13:17 kind-control-plane containerd[25550]: time="2020-07-13T09:13:17.696627579Z" level=info msg="TearDown network for sandbox \"b78efe047b461be029f0b2e8a08a1f478bfaee97a57c253945e5ca4289e38014\" successfully"
Jul 13 09:13:17 kind-control-plane containerd[25550]: time="2020-07-13T09:13:17.861819714Z" level=info msg="TearDown network for sandbox \"b78efe047b461be029f0b2e8a08a1f478bfaee97a57c253945e5ca4289e38014\" successfully"
Jul 13 09:13:30 kind-control-plane containerd[25550]: time="2020-07-13T09:13:30.861008355Z" level=info msg="TearDown network for sandbox \"b78efe047b461be029f0b2e8a08a1f478bfaee97a57c253945e5ca4289e38014\" successfully"

full log here
https://pastebin.com/9d83jMvj

if we log the number of iptables called issued using a tool like execsnoop, we can see that the portmap plugin executes multiples iptables and ip6tables calls using the --wait flag without any timeout, that means wait forever (ref coreos/go-iptables#75)

sudo ./execsnoop.bt | grep -E ip6?tables

84112      78102 /usr/sbin/iptables --version
84113      78103 /usr/sbin/iptables -t nat -S OUTPUT --wait
84115      78104 /usr/sbin/ip6tables --version
84116      78105 /usr/sbin/ip6tables -t nat -S OUTPUT --wait
84117      78106 /usr/sbin/iptables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
84118      78107 /usr/sbin/iptables -t nat -F CNI-DN-9d924da12ab9b7dd58f9a --wait
84119      78108 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-DNAT --wait
84121      78109 /usr/sbin/iptables -t nat -D CNI-HOSTPORT-DNAT -p tcp -m comment --comment dnat name: "kindnet" id: "034ae066e50dbfd10de36326a95ce2ac23276958e2408c02eb5a3c56732b6574" -m multiport --dports 8080 -j
84125      78110 /usr/sbin/iptables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
84128      78111 /usr/sbin/iptables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
84130      78112 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-SNAT --wait
84131      78113 /usr/sbin/iptables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
84132      78114 /usr/sbin/ip6tables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
84133      78115 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-DNAT --wait
84134      78116 /usr/sbin/ip6tables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
84136      78117 /usr/sbin/ip6tables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
84137      78118 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-SNAT --wait
84138      78119 /usr/sbin/ip6tables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
84250      78155 /usr/sbin/iptables --version
84252      78156 /usr/sbin/iptables -t nat -S OUTPUT --wait
84253      78157 /usr/sbin/ip6tables --version
84254      78158 /usr/sbin/ip6tables -t nat -S OUTPUT --wait
84255      78159 /usr/sbin/iptables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
84256      78160 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-DNAT --wait
84257      78161 /usr/sbin/iptables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
84258      78162 /usr/sbin/iptables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
84259      78163 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-SNAT --wait
84260      78164 /usr/sbin/iptables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
84261      78165 /usr/sbin/ip6tables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
84263      78166 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-DNAT --wait
84264      78167 /usr/sbin/ip6tables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
84265      78168 /usr/sbin/ip6tables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
84266      78169 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-SNAT --wait
84267      78170 /usr/sbin/ip6tables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
85250      78243 /usr/sbin/iptables --version
85251      78244 /usr/sbin/iptables -t nat -S OUTPUT --wait
85253      78245 /usr/sbin/ip6tables --version
85253      78246 /usr/sbin/ip6tables -t nat -S OUTPUT --wait
85255      78247 /usr/sbin/iptables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
85257      78248 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-DNAT --wait
85260      78250 /usr/sbin/iptables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
85262      78251 /usr/sbin/iptables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
85264      78252 /usr/sbin/iptables -t nat -S CNI-HOSTPORT-SNAT --wait
85265      78253 /usr/sbin/iptables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
85266      78254 /usr/sbin/ip6tables -t nat -N CNI-DN-9d924da12ab9b7dd58f9a --wait
85267      78255 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-DNAT --wait
85268      78256 /usr/sbin/ip6tables -t nat -X CNI-DN-9d924da12ab9b7dd58f9a --wait
85269      78257 /usr/sbin/ip6tables -t nat -N CNI-SN-9d924da12ab9b7dd58f9a --wait
85270      78258 /usr/sbin/ip6tables -t nat -S CNI-HOSTPORT-SNAT --wait
85271      78259 /usr/sbin/ip6tables -t nat -X CNI-SN-9d924da12ab9b7dd58f9a --wait
88303      78290 /usr/sbin/iptables -t nat -S --wait

xref: kubernetes/kubernetes#92811 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

portmap should not perform deletions if not portMapping config received aojea/plugins
1 participant
@aojea