Make the iptables FORWARD chain rule additions optional

[KashifSaadat]

Expected Behavior

1. Configure the setting FELIX_CHAININSERTMODE to append in the canal manifest file
2. Deploy to a node
3. Inspect iptables: iptables -nv -L FORWARD
4. I would expect only 1 rule to be inserted by calico (at the bottom of the chain, if you have inserted custom iptables rules). For example:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     tcp  --  cali+  *       0.0.0.0/0            169.254.169.254      tcp dpt:80 state NEW 
    0     0 cali-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wUHhoiAYhphO9Mso */

Current Behavior

Flannel v0.9.1 includes a change to add 2 rules to the FORWARD chain (#872). When Calico is configured with append mode, the Calico rule is added to the bottom of the chain, resulting in the following rule order:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     tcp  --  cali+  *       0.0.0.0/0            169.254.169.254      tcp dpt:80 state NEW reject-with icmp-port-unreachable
    0     0 ACCEPT     all  --  *      *       10.10.0.0/16         0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.10.0.0/16
    0     0 cali-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wUHhoiAYhphO9Mso */

Because of this, the cali-FORWARD rule is never hit and all traffic in/out is accepted by default. This means that k8s ingress/egress network policies will have no effect.

You won't encounter this issue if the chain insert mode isn't set to append (it is insert by default). Flannel will still add 2 rules to the forward chain, but they will never be hit because the calico rule is processed first.

Possible Solution

One suggestion would be to provide a flag to flannel such as "AddIPTablesForwardRules": false.

Context

I'm performing these deployments using kops v1.8.0, and the relevant manifest file is located here: https://github.com/kubernetes/kops/blob/1.8.0/upup/models/cloudup/resources/addons/networking.projectcalico.org.canal/k8s-1.8.yaml.template

I set FELIX_CHAININSERTMODE to append so that I can insert rules at the top of the INPUT and FORWARD chains to process specific global REJECT rules before the network policies take effect (preventing pods communicating with ETCD API, Kube API, AWS meta-data endpoint, etc).

Your Environment

• Calico version: v2.6.2
• Flannel version: v0.9.1 (the issue was introduced in this version)
• Kops version: v1.8.0 (flannel has been downgraded to v0.9.0 for the previous functionality to continue working)
• Operating System and version: CoreOS stable v1520.8.0

CC @tomdee

Related issues:

• Flannel v0.9.1 adds FORWARD rules breaking networkpolicy behaviour projectcalico/canal#115
• Canal on K8s v1.8 with chain append mode no longer respects networkpolicies kubernetes/kops#4037

[KashifSaadat]

Any thoughts on this?

[chenchun]

+1 I'm trying to implement network policy in my CNI plugin and having the same problem.

[KashifSaadat]

Fixed in Release v0.11.0

Set FLANNELD_IPTABLES_FORWARD_RULES to false for those default ACCEPT rules to no longer be appended to the iptables FORWARD chain.