VPP neighbor table is not updated properly in case of IP or MAC address change

[szvincze]

When we use an OVS interface on the host, the IP and MAC address can change at worker node restart.
In such cases it will not stored in VPP neighbor table which causes issues at restart (address resolution does not work).

It occurs with NSM version 1.6.2 and 1.8.0 too.

[ljkiraly]

Reproduction steps:

• run Kernel2Ethernet2Kernel example use-case in kind

• check that the ping works between NSC and NSE

• check the neighbor table inside vpp

kubectl get pods -o wide -n nsm-system -l app=forwarder-vpp
NAME                  READY   STATUS    RESTARTS      AGE   IP           NODE           NOMINATED NODE   READINESS GATES
forwarder-vpp-2g4dh   1/1     Running   1 (27m ago)   34m   172.18.0.3   kind-worker2   <none>           <none>
forwarder-vpp-stvwh   1/1     Running   1 (25m ago)   34m   172.18.0.2   kind-worker    <none>           <none>

kubectl exec -n nsm-system forwarder-vpp-2g4dh vppctl sh ip neighbor
    Time                       IP                    Flags      Ethernet              Interface       
      4.5853               172.18.0.1                  D    02:42:3e:b4:19:e3 host-eth0
      1.2279               172.18.0.2                  S    02:42:ac:12:00:02 host-eth0
      1.2277               172.18.0.4                  S    02:42:ac:12:00:04 host-eth0

• set the hardware address of the interface used for vxlan connection in kind-worker node

> docker exec kind-worker ip a s dev eth0
231: eth0@if232: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::2/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:2/64 scope link 
       valid_lft forever preferred_lft forever

> docker exec kind-worker ip link set dev eth0 addr 02:42:ac:12:00:22
> docker exec kind-worker ip link show dev eth0
231: eth0@if232: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

• restart the forwarder running in this worker node:

> docker exec -it kind-worker pkill vpp

• check the neighbor table (remains unchanged)

kubectl exec -n nsm-system forwarder-vpp-2g4dh vppctl sh ip neighbor
    Time                       IP                    Flags      Ethernet              Interface       
      4.5853               172.18.0.1                  D    02:42:3e:b4:19:e3 host-eth0
      1.2279               172.18.0.2                  S    02:42:ac:12:00:02 host-eth0
      1.2277               172.18.0.4                  S    02:42:ac:12:00:04 host-eth0

• check that the ping does not work between NSC and NSE

• removing the old neighbor and adding back the right one to vpp by hand can solve the problem (after this the ping works again)

kubectl exec -n nsm-system forwarder-vpp-2g4dh vppctl ip neighbor del host-eth0 172.18.0.2 02:42:ac:12:00:02
kubectl exec -n nsm-system forwarder-vpp-2g4dh vppctl ip neighbor host-eth0 172.18.0.2 02:42:ac:12:00:22
kubectl exec -n nsm-system forwarder-vpp-2g4dh vppctl sh ip neighbor
 
    273.3889               172.18.0.1                  D    02:42:3e:b4:19:e3 host-eth0
    379.4712               172.18.0.2                  D    02:42:ac:12:00:22 host-eth0
    1.2277               172.18.0.4                  S    02:42:ac:12:00:04 host-eth0

What would be the side effect of setting dynamic neighbors (instead of static) at interface creation when initializing the forwarder?

[glazychev-art]

@ljkiraly
Thanks for the detailed steps!
Do dynamic neighbors help you?
I don't think there are any side effects. I tested it on a couple of clusters - everything works well

[ljkiraly]

@glazychev-art Yes, the dynamic neighbor setup helps. Thanks.

[zolug]

Hi @glazychev-art,

Could you please explain, how VPP is supposed to learn about neighbors?
Based on your linked PR vpp-forwarder on startup fetches the reachable neighbor's list, and configures them as neighbors in VPP.
How else, can VPP learn about neighbors? (For example in case a new node is introduced into the system.)

I'm curious because in one of our system (dualstack-cluster with IPv6 "primary" IPs), one VPP instance started by the forwarder was not informed about a certain neighbor at start. So, the missing entry is not visible with the static flag among the the neighbors in vppctl. However, there's also no dynamic entry for the particular IP. And it actually causes traffic problems because vxlan encapsulation fails. Tracing shows drop with ip6-discover-neighbor: address overflow drops

Thanks in advance.

[zolug]

Hi @glazychev-art,

Could you please explain, how VPP is supposed to learn about neighbors? Based on your linked PR vpp-forwarder on startup fetches the reachable neighbor's list, and configures them as neighbors in VPP. How else, can VPP learn about neighbors? (For example in case a new node is introduced into the system.)

I'm curious because in one of our system (dualstack-cluster with IPv6 "primary" IPs), one VPP instance started by the forwarder was not informed about a certain neighbor at start. So, the missing entry is not visible with the static flag among the the neighbors in vppctl. However, there's also no dynamic entry for the particular IP. And it actually causes traffic problems because vxlan encapsulation fails. Tracing shows drop with ip6-discover-neighbor: address overflow drops

Thanks in advance.

I think the ingress ACL filters set by the forwarder might be wrong for IPv6. Instead of ICMPv6 Type the Code fields are set: https://github.com/networkservicemesh/cmd-forwarder-vpp/blob/main/internal/vppinit/acl.go#L111
Most probably causing the Neighbor Advertisement to get dropped in my case. I will build a new forwarder where ingress sets the ICMPv6 Types to verify it.

Packet 41

06:31:58:815638: af-packet-input
  af_packet: hw_if_index 2 next-index 4
    tpacket2_hdr:
      status 0x20000005 len 86 snaplen 86 mac 66 net 80
      sec 0x645130d0 nsec 0x12f3d524 vlan 0 vlan_tpid 0
06:31:58:815641: ethernet-input
  IP6: fa:16:3e:a9:a5:9f -> fa:16:3e:d7:89:d1
06:31:58:815642: ip6-input
  ICMP6: fd08::d -> fd08::21
    tos 0x00, flow label 0x0, hop limit 255, payload length 32
06:31:58:815643: acl-plugin-in-ip6-fa
  acl-plugin: lc_index: 1, sw_if_index 2, next index 0, action: 0, match: acl 0 rule 3 trace_bits 00000000
  pkt info 00000000000008fd 0d00000000000000 00000000000008fd 2100000000000000 0002033a00000088 0a00ffff00000001
   lc_index 1 l3 ip6 fd08::d -> fd08::21 l4 lsb_of_sw_if_index 2 proto 58 l4_is_input 1 l4_slow_path 1 l4_flags 0x03
port 136 -> 0 tcp flags (invalid) 00 rsvd 0
06:31:58:815646: error-drop
  rx:host-eth0
06:31:58:815647: drop
  acl-plugin-in-ip6-fa: ACL deny packets
  

vpp# show acl-plugin acl
acl-index 0 count 4 tag {nsm-vppinit-denyall-ingress}
          0: ipv6 permit src ::/0 dst ::/0 proto 58 sport 0 dport 134
          1: ipv6 permit src ::/0 dst ::/0 proto 58 sport 0 dport 136
          2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
          3: ipv6 deny src ::/0 dst ::/0 proto 0 sport 0 dport 0
  applied inbound on sw_if_index: 2
  applied outbound on sw_if_index:
  used in lookup context index: 1
acl-index 1 count 4 tag {nsm-vppinit-denyall-egress}
          0: ipv4 permit src 0.0.0.0/0 dst 0.0.0.0/0 proto 58 sport 133 dport 0
          1: ipv6 permit src ::/0 dst ::/0 proto 58 sport 135 dport 0
          2: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
          3: ipv4 deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
  applied inbound on sw_if_index:
  applied outbound on sw_if_index: 2
  used in lookup context index: 0
acl-index 2 count 1 tag {nsm-pinhole port 4789}
          0: ipv6 permit src ::/0 dst fd08::21/128 proto 17 sport 0-65535 dport 4789
  applied inbound on sw_if_index: 2
  applied outbound on sw_if_index:
  used in lookup context index: 1
acl-index 3 count 1 tag {nsm-pinhole port 4789}
          0: ipv6 permit src fd08::21/128 dst ::/0 proto 17 sport 0-65535 dport 4789
  applied outbound on sw_if_index: 2
  used in lookup context index: 0

[zolug]

Hi @glazychev-art,

I fixed the IPv6 ingress ACL rules to use Type instead of Code fields. With those changes, ICMPv6 Neighbor Advertisements were accepted, and dynamic neighbor resolution started to work.
I will create a separate issue and PR for better tracking.

[glazychev-art]

Hi @zolug
Good catch, thank you!

I think we can close this issue