New 0.30.x ACL features break iptables-nft routing table

[Spiritreader]

Describe the problem

It is no longer possible to list the routing table when netbird is active because it inserts incompatible rules.
I cannot fully switch over to nftables because docker requires iptables and modifying existing chains via nft is discouraged.

There are also side effects where you cannot insert certain rules anymore (which affects services like docker when it is creating new networks as it needs to modify iptables rules). Some iptables insertion and delete commands depend on the table being parsable.

To Reproduce

While netbird is running

❯ sudo iptables -L
iptables v1.8.9 (nf_tables): table `filter' is incompatible, use 'nft' tool.

❯ netbird down
Disconnected

❯ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (3 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp dpt:http
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:http
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:https

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Expected behavior

iptables work as expected (before 0.30).
Netbird does not modify any tables managed by iptables-nft via nftables.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft. won't work as outlined by #2725 (comment)

Are you using NetBird Cloud?

selfhosted

NetBird version

0.30.1

Do you face any (non-mobile) client issues?

The aforementioned issue affects linux clients with iptables installed.

[MDMeridio001]

I'm also having the same issue on version 0.30.0.
iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool.

[Spiritreader]

Yeah, I'm also not quite sure which rule is causing the issue. I would assume any rule that is inserted into that chain via nftables causes the issue?

❯ sudo nft list table filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain DOCKER {
                iifname != "br-2cee86ba52c6" oifname "br-2cee86ba52c6" ip daddr 172.19.0.2 tcp dport 80 counter packets 0 bytes 0 accept
                iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 80 counter packets 0 bytes 0 accept
                iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 443 counter packets 5 bytes 260 accept
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 jump DOCKER-ISOLATION-STAGE-2
                counter packets 7013 bytes 18823363 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
                oifname "br-4018d9fe98eb" counter packets 0 bytes 0 drop
                oifname "br-2cee86ba52c6" counter packets 0 bytes 0 drop
                counter packets 2925 bytes 2719110 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept
                iifname "wt0" counter packets 124650 bytes 227839202 accept
                counter packets 7013 bytes 18823363 jump DOCKER-USER
                counter packets 7013 bytes 18823363 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
                oifname "br-4018d9fe98eb" ct state related,established counter packets 3186 bytes 7314863 accept
                oifname "br-4018d9fe98eb" counter packets 5 bytes 260 jump DOCKER
                iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 accept
                iifname "br-4018d9fe98eb" oifname "br-4018d9fe98eb" counter packets 0 bytes 0 accept
                oifname "br-2cee86ba52c6" ct state related,established counter packets 897 bytes 8789130 accept
                oifname "br-2cee86ba52c6" counter packets 0 bytes 0 jump DOCKER
                iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 accept
                iifname "br-2cee86ba52c6" oifname "br-2cee86ba52c6" counter packets 0 bytes 0 accept
        }

        chain DOCKER-USER {
                counter packets 7013 bytes 18823363 return
        }
}

It might be necessary to insert those rules

oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept
iifname "wt0" counter packets 124650 bytes 227839202 accept

via iptables, something like

-A FORWARD -i wt0 -m state -j ACCEPT
-A FORWARD -o wt0 -m state --state RELATED,ESTABLISHED -j ACCEPT

instead of using nft add ...

[horzadome]

FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them.
I have no idea how some of them were unaffected.
The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled.
But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error.
Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again.

[Spiritreader]

FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them. I have no idea how some of them were unaffected. The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled. But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error. Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again.

Oh you're right, it's no longer possible to add certain rules that rely on the table being parsable, I'll adjust the title. I did encounter that but since I didn't make any changes to docker yet after updating to 0.30, I haven't noticed so far.

As a workaround in the meantime if you don't want to downgrade, you can try running the netbird client in a docker container via network_mode=host
However that will come with added configuration to make DNS function correctly. That's what I'm doing on some of my machines and the host iptables remain intact this way.

[lixmal]

This will be fixed in #2727

Meanwhile you can try setting env NB_SKIP_NFTABLES_CHECK=true to use iptables only.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft.

The latter is not feasible since an accept verdict won't stop the iptables-nft chain from being traversed, unlike a drop verdict. (We already have a filter chain in table netbird)

The culprit is -o wt0 -m state --state RELATED,ESTABLISHED, which must be a bug since the iptables-translate emits the same iptables-nft incompatible rule

[MDMeridio001]

This will be fixed in #2727

Meanwhile you can try setting env NB_SKIP_NFTABLES_CHECK=true to use iptables only.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft.

The latter is not feasible since an accept verdict won't stop the iptables-nft chain from being traversed, unlike a drop verdict. (We already have a filter chain in table netbird)

The culprit is -o wt0 -m state --state RELATED,ESTABLISHED, which must be a bug since the iptables-translate emits the same iptables-nft incompatible rule

I've tried the suggested workaround but I still get the same error from iptables.

❯ netbird down
Disconnected

❯ export NB_SKIP_NFTABLES_CHECK=true

❯ netbird up
Connected

❯ sudo iptables -L
iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool.

[lixmal]

The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd

[MDMeridio001]

The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd

Ok, can confirm that it works.

❯ sudo mkdir -p /etc/sysconfig

❯ echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird
NB_SKIP_NFTABLES_CHECK=true

❯ sudo systemctl restart netbird

[Spiritreader]

Fixed in 0.30.2

I un-set the abovementioned environment variable, and restarted the netbird service.
The iptables filter table now remains functional and the rest of the rules show up correctly if queried via nft.

Thanks a lot for the quick response and implementation!!