Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NAT stop working for incomming connections after restart #43

Closed
VenusGen opened this issue Nov 17, 2019 · 11 comments
Closed

NAT stop working for incomming connections after restart #43

VenusGen opened this issue Nov 17, 2019 · 11 comments

Comments

@VenusGen
Copy link

VenusGen commented Nov 17, 2019 •
edited
Loading

I wrote Caddy and ipv6nat in a docker-compose file, It works very well. But after I executed the docker-compose restart command, I am still able to ping other v6 addresses from the Caddy container but can't receive any more incoming requests (It was able to receive when userland-proxy enabled, but Caddy couldn't get the real IP address).

I think it's probably similar to #14 , probably we need to remove iptables when container stopping?

ip6tables -L:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (5 references)
target     prot opt source               destination         
ACCEPT     tcp      anywhere             fd00:beef::3         tcp dpt:https
ACCEPT     tcp      anywhere             fd00:beef::3         tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all      anywhere             anywhere

docker-compose.yml:

version: '2.1'
services:
  caddy:
    image: abiosoft/caddy
    container_name: caddy
    ports:
      - "80:80"
      - "443:443"
    environment:
      - CADDYPATH=/caddy
      - ACME_AGREE=true
    volumes:
      - ./Caddyfile:/etc/Caddyfile
      - caddyacme:/caddy/acme
    restart: always
    networks:
      v6net:
    depends_on:
      - ipv6nat
  ipv6nat:
    image: robbertkl/ipv6nat
    restart: always
    network_mode: "host"
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
networks:
  v6net:
    driver: bridge
    enable_ipv6: true
    ipam:
      driver: default
      config:
      - subnet: 172.20.0.0/16
        gateway: 172.20.0.1
      - subnet: fd00:beef::/80
volumes:
  caddyacme:

Docker Environment:

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Docker Compose:

docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018
@bephinix
Copy link
Contributor

@VenusGen You should use the -cleanup flag if you start docker-ipv6nat. Otherwise there might be duplicated or intefering rules if you restart docker-ipv6nat. Can you test this?

@VenusGen
Copy link
Author

It seems avoided duplicated rules now, but when it clear up the rules, it also wiped the 443 & 80 mapping...

➜  ip6tables -L                               
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

**Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp      anywhere             fd00:beef::3         tcp dpt:https
ACCEPT     tcp      anywhere             fd00:beef::3         tcp dpt:http**

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all      anywhere             anywhere            
➜  docker-compose restart                     
Restarting caddy           ... done
Restarting ipv6nat         ... done
➜  ip6tables -L          
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all      anywhere             anywhere            
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all      anywhere             anywhere

@robbertkl
Copy link
Owner

Could you please use the output of ip6tables-save instead of ip6tables -L? This would give us the full rules.

Not sure why the 80/443 rules are missing. Is your docker-compose.yml still like in your first post? Could you run ipv6nat with the -debug flag and take a look at the output?

@VenusGen
Copy link
Author

Is your docker-compose.yml still like in your first post?

No, I add a line command: --retry -cleanup -debug after the volumes part of ipv6nat to run with those flags.

The full logs are attached below. Briefly, the log shows v6nat removed 443&80 but never add them back.

➜  ip6tables-save
# Generated by ip6tables-save v1.6.1 on Mon Nov 18 09:39:57 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d ::1/128 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o br-0d708b1e1de8 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-0d708b1e1de8 -j MASQUERADE
-A POSTROUTING -s fd00:beef::3/128 -d fd00:beef::3/128 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s fd00:beef::3/128 -d fd00:beef::3/128 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -o br-1aa0f87300b3 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-1aa0f87300b3 -j MASQUERADE
-A POSTROUTING -o br-50ddbadc04b4 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-50ddbadc04b4 -j MASQUERADE
-A POSTROUTING -o br-c11ddedb2910 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-c11ddedb2910 -j MASQUERADE
-A POSTROUTING -o br-e59b5f972548 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-e59b5f972548 -j MASQUERADE
-A POSTROUTING -o br-37f1ec58650a -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-37f1ec58650a -j MASQUERADE
-A POSTROUTING -s fd00:beef::2/128 -d fd00:beef::2/128 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s fd00:beef::2/128 -d fd00:beef::2/128 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -o br-1b81271987dc -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-1b81271987dc -j MASQUERADE
-A DOCKER -p tcp -m tcp --dport 443 -j DNAT --to-destination [fd00:beef::3]:443
-A DOCKER -p tcp -m tcp --dport 80 -j DNAT --to-destination [fd00:beef::3]:80
COMMIT
# Completed on Mon Nov 18 09:39:57 2019
# Generated by ip6tables-save v1.6.1 on Mon Nov 18 09:39:57 2019
*filter
:INPUT ACCEPT [10:576]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [6:520]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-0d708b1e1de8 -j DOCKER
-A FORWARD -o br-0d708b1e1de8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j ACCEPT
-A FORWARD -i br-0d708b1e1de8 -o br-0d708b1e1de8 -j ACCEPT
-A DOCKER -d fd00:beef::3/128 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d fd00:beef::3/128 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-0d708b1e1de8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Nov 18 09:39:57 2019

➜  docker-compose restart                     
Restarting caddy   ... done
Restarting ipv6nat ... done

➜  ip6tables-save        
# Generated by ip6tables-save v1.6.1 on Mon Nov 18 09:40:21 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d ::1/128 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o br-0d708b1e1de8 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-0d708b1e1de8 -j MASQUERADE
-A POSTROUTING -o br-1aa0f87300b3 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-1aa0f87300b3 -j MASQUERADE
-A POSTROUTING -o br-50ddbadc04b4 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-50ddbadc04b4 -j MASQUERADE
-A POSTROUTING -o br-c11ddedb2910 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-c11ddedb2910 -j MASQUERADE
-A POSTROUTING -o br-e59b5f972548 -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-e59b5f972548 -j MASQUERADE
-A POSTROUTING -o br-37f1ec58650a -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-37f1ec58650a -j MASQUERADE
-A POSTROUTING -s fd00:beef::2/128 -d fd00:beef::2/128 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s fd00:beef::2/128 -d fd00:beef::2/128 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -o br-1b81271987dc -m addrtype --dst-type LOCAL -j MASQUERADE
-A POSTROUTING -s fd00:beef::/80 ! -o br-1b81271987dc -j MASQUERADE
COMMIT
# Completed on Mon Nov 18 09:40:21 2019
# Generated by ip6tables-save v1.6.1 on Mon Nov 18 09:40:21 2019
*filter
:INPUT ACCEPT [4:224]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [16:1568]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-0d708b1e1de8 -j DOCKER
-A FORWARD -o br-0d708b1e1de8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j ACCEPT
-A FORWARD -i br-0d708b1e1de8 -o br-0d708b1e1de8 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-0d708b1e1de8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Nov 18 09:40:21 2019

➜  docker-compose logs   
Attaching to caddy, ipv6nat
ipv6nat  | # Warning: iptables-legacy tables present, use iptables-legacy to see them
ipv6nat  | 2019/11/18 09:37:50 docker-ipv6nat is running in debug mode
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -j RETURN
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -j RETURN
ipv6nat  | 2019/11/18 09:37:50 rule added: -t nat -A PREROUTING 1 -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | 2019/11/18 09:37:50 rule added: -t nat -A OUTPUT 1 -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 1 -j DOCKER-ISOLATION-STAGE-1
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 1 -j DOCKER-USER
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 3 -o br-0d708b1e1de8 -j DOCKER
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 4 -o br-0d708b1e1de8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 5 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A FORWARD 6 -i br-0d708b1e1de8 -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:37:50 rule added: -t nat -A POSTROUTING 1 -s fd00:beef::/80 ! -o br-0d708b1e1de8 -j MASQUERADE
ipv6nat  | 2019/11/18 09:37:50 rule added: -t nat -A POSTROUTING 1 -o br-0d708b1e1de8 -m addrtype --dst-type LOCAL -j MASQUERADE
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o br-0d708b1e1de8 -j DROP
ipv6nat  | 2019/11/18 09:37:50 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j DOCKER-ISOLATION-STAGE-2
ipv6nat  | 2019/11/18 09:37:53 rule added: -t filter -A DOCKER 1 -d fd00:beef::3 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 443 -j ACCEPT
ipv6nat  | 2019/11/18 09:37:53 rule added: -t nat -A POSTROUTING 3 -s fd00:beef::3 -d fd00:beef::3 -p tcp -m tcp --dport 443 -j MASQUERADE
ipv6nat  | 2019/11/18 09:37:53 rule added: -t nat -A DOCKER 1 -d 0/0 -p tcp -m tcp --dport 443 -j DNAT --to-destination [fd00:beef::3]:443
ipv6nat  | 2019/11/18 09:37:53 rule added: -t filter -A DOCKER 2 -d fd00:beef::3 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 80 -j ACCEPT
ipv6nat  | 2019/11/18 09:37:53 rule added: -t nat -A POSTROUTING 4 -s fd00:beef::3 -d fd00:beef::3 -p tcp -m tcp --dport 80 -j MASQUERADE
ipv6nat  | 2019/11/18 09:37:53 rule added: -t nat -A DOCKER 2 -d 0/0 -p tcp -m tcp --dport 80 -j DNAT --to-destination [fd00:beef::3]:80
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER -d fd00:beef::3 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 443 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D POSTROUTING -s fd00:beef::3 -d fd00:beef::3 -p tcp -m tcp --dport 443 -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D DOCKER -d 0/0 -p tcp -m tcp --dport 443 -j DNAT --to-destination [fd00:beef::3]:443
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER -d fd00:beef::3 ! -i br-0d708b1e1de8 -o br-0d708b1e1de8 -p tcp -m tcp --dport 80 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D POSTROUTING -s fd00:beef::3 -d fd00:beef::3 -p tcp -m tcp --dport 80 -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D DOCKER -d 0/0 -p tcp -m tcp --dport 80 -j DNAT --to-destination [fd00:beef::3]:80
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER-ISOLATION-STAGE-1 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j DOCKER-ISOLATION-STAGE-2
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER-ISOLATION-STAGE-2 -o br-0d708b1e1de8 -j DROP
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D FORWARD -o br-0d708b1e1de8 -j DOCKER
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D FORWARD -o br-0d708b1e1de8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D FORWARD -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D FORWARD -i br-0d708b1e1de8 -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D POSTROUTING -o br-0d708b1e1de8 -m addrtype --dst-type LOCAL -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D POSTROUTING -s fd00:beef::/80 ! -o br-0d708b1e1de8 -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D FORWARD -j DOCKER-ISOLATION-STAGE-1
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER-ISOLATION-STAGE-1 -j RETURN
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t filter -D DOCKER-ISOLATION-STAGE-2 -j RETURN
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | 2019/11/18 09:40:09 rule removed: -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | # Warning: iptables-legacy tables present, use iptables-legacy to see them
ipv6nat  | 2019/11/18 09:40:10 docker-ipv6nat is running in debug mode
ipv6nat  | 2019/11/18 09:40:10 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -j RETURN
ipv6nat  | 2019/11/18 09:40:10 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -j RETURN
ipv6nat  | 2019/11/18 09:40:10 rule added: -t nat -A PREROUTING 1 -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | 2019/11/18 09:40:10 rule added: -t nat -A OUTPUT 1 -m addrtype --dst-type LOCAL -j DOCKER
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 1 -j DOCKER-ISOLATION-STAGE-1
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 1 -j DOCKER-USER
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 3 -o br-0d708b1e1de8 -j DOCKER
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 4 -o br-0d708b1e1de8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 5 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A FORWARD 6 -i br-0d708b1e1de8 -o br-0d708b1e1de8 -j ACCEPT
ipv6nat  | 2019/11/18 09:40:11 rule added: -t nat -A POSTROUTING 1 -s fd00:beef::/80 ! -o br-0d708b1e1de8 -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:11 rule added: -t nat -A POSTROUTING 1 -o br-0d708b1e1de8 -m addrtype --dst-type LOCAL -j MASQUERADE
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o br-0d708b1e1de8 -j DROP
ipv6nat  | 2019/11/18 09:40:11 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i br-0d708b1e1de8 ! -o br-0d708b1e1de8 -j DOCKER-ISOLATION-STAGE-2

@robbertkl
Copy link
Owner

So if I see it correctly, it does add the rules correctly (09:37), then removes them correctly (09:40) on quit (when restarting) but does not add them on the 2nd start (09:40, when restarting).

How did you start it the first time? (09:37) Did you start only ipv6nat at that time? Perhaps it could be some kind of race condition when restarting ipv6nat and caddy at the same time.

Could you also test first restarting caddy, then ipv6nat (instead of restarting them at the same time) and also first ipv6nat, then caddy. Just wondering which cases will end up with the correct rules.

@VenusGen
Copy link
Author

The only difference is in the 1st start I used docker-compose down && docker-compose up -d to destroy whole things and rebuilt them, and in the 2nd I used docker-compose restart.

Caddy should start later than v6nat cause I set depends_on: v6nat for caddy, but not sure if docker will follow this when using restart command.

@VenusGen
Copy link
Author

VenusGen commented Nov 18, 2019 •
edited
Loading

If I restart them separately, whatever the order are, iptables keeps correct ( with full records include 80&443)

@VenusGen
Copy link
Author

Docker Compose seems don't guarantee the sequence of restart:

https://stackoverflow.com/questions/49063920/docker-service-restart-without-order-sequence-when-using-docker-compose-depends

https://docs.docker.com/compose/startup-order/

@VenusGen
Copy link
Author

If I restart them separately, whatever the order are, iptables keeps correct ( with full records include 80&443)

Actually they are same, whenever I restart v6nat, caddy is running normally at these two situations.

@nim65s
Copy link

nim65s commented May 14, 2020

Thanks for the tips. At some point, my X-Forwarded-For started showing a docker IPv4 when the request was made in IPv6 again, as if I just didn't had my ipv6nat container up.

I did a docker-compose down on my docker-file with traefik & ipv6nat, commented out ipv6nat, ran docker-compose up, uncommented ipv6nat, and ran docker-compose up again, and now it works :)

@bephinix
Copy link
Contributor

@robbertkl Can we close this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nim65s @robbertkl @bephinix @VenusGen