1st fragment dropped on VxLAN connection

[LionelJouin]

Current Behavior

When pinging or sending UDP packets from the NSE to the NSC through a NSM VxLAN link (created by the vpp-forwarder) with a packet size higher than the MTU, the NSC does not receive the first fragment.

Failure Information (for bugs)

We checked the vxlan packets on both vpp forwarders (so on the 2 nodes), and we could see the 2 fragments are correctly sent on the NSE node and correctly received on the NSC node.
In the vpp-forwarder which runs on the same node as the NSC, we could see the drop counter of the tap interface corresponding to the NSC one is increasing each time we are sending fragmented traffic.

Steps to Reproduce

• Create a kernel to vxlan to kernel connection with a NSE and a NSC on different nodes.
• Connect to the NSC, get its NSM IP and run tcpdump: tcpdump -nnvvXSu -i any icmp
• Connect to the NSE and ping the NSC IP with a packet size higher than the MTU of the NSM interface. For example, the MTU of our interface was 1500, so I set the packet size to 2000. ping -s 2000 169.254.0.3
• On the output of tcpdump in the NSC, you should see it has received only the 2nd fragment and no response is sent to the NSE

Context

Versions:

• Kubernetes: 1.20.2
• forwarder-vpp=cd71461
• nsmgr=505add9
• registry-memory=c098147
• nse-icmp-responder-vpp=40a2d83
• nsc-vpp=68aa825

I tried with v1.0.0 images and I could not see this issue. It was working fine

[denis-tingaikin]

1. Could you please test it with client from https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/webhook#client-requests-for-postgresql-service
2. Could you check is problem also reproducing on NSE side?

[LionelJouin]

I confirm this issue happens in both way (sending packet to the NSE from the NSC, and sending packet to the NSC from the NSE)

[LionelJouin]

I also tried with with client from https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/webhook#client-requests-for-postgresql-service
I have the same issue.

[edwarnicke]

@NikitaSkrynnik Some things to look at here as you debug:

You can kubctl into the forwarder pod and use vppctl to trace the packets through vpp.

If I understand @LionelJouin correclty, the ping from either the NSC or NSE is going something like this:

NSC ---- kernel --- VPP (in forwarder-1) ---vxlan --- VPP (in forwarder-2) --- kernel --- NSE

If you run ping in the NSC, you would run the trace in the VPP in forwarder-1.

I suspect the things you want to trace are:

vppctl trace add af-packet-input
vppctl trace add tuntap-rx
vppctl trace add virtio-input

Set these before you send the ping from the NSC.

That should give you a bit more information about what may be happening inside VPP.

@LionelJouin Feel free to try that too and report your findings back here :)

[edwarnicke]

@LionelJouin Do you happen to know if vethpairs or tapv2 is being used in the setup where you see the issue?

Ed

[edwarnicke]

@LionelJouin If you are using vethpairs... I have a potential root cause identified.

With a vethpair, you have two sides of the pair, for the sake of conversation lets call them (nsc-1, vpp-1). Currently it appears that nsc-1 is having its mtu set correctly, but vpp-1 is not. Further, host-vpp-1 interface in vpp has its mtu set to match nsc-1. (note: this is all the same as well on the NSE side).

The result is that if I send a packet into nsc-1 that is fine for the mtu on nsc-1, but to small for the mtu on vpp-1 ... its dropped by the kernel.

If a packet arrives and is sent out host-vpp-1 and its smaller than the mtu on host-vpp-1 but larger than the mtu on vpp-1... the kernel drops it.

My (tentative) theory as to why you are seeing the second fragment and not the first is that you are properly getting fragmentation of the original packet at its origin... but along the way we are hitting this issue with the first fragment (which is larger) causing it to get dropped. The second smaller fragment is sufficiently small it doesn't hit this issue.

Could you check to see if that's what you are seeing?

Steps would include:

1. Checking to see if you are using vethpairs + af-packet to connect kernel interfaces to vpp. Some ways to check this is that af-packet interfaces have the form 'host-${kernel interface name}'. Another way to check is that if you use ip addr in the name space the kernel interface is is, you should see something like '${interface name}@{vethpeer interfacename' in the output.
2. Check the MTUs of both ends of the veth pairs and the 'host' interfaces in vpp and report them here. It would help a lot in figuring this out.
3. If you are cross connecting to something like vxlan or wireguard, if you could report the mtus for those and the mtus for the underlying 'node' interface they are running across (both in vpp and on the host) that would help. I'm pretty sure the bug is isolated to vethpairs+af-packet ... but want to be certain.

Note: to get mtu info in vpp:

vppctl show int

[LionelJouin]

I tried the trace commands, but I don't think I found anything relevant.

I use VxLAN, and I have these MTUs:

• underlying 'node' interface (eth0) ; MTU = 1500
• VPP interface host-eth0 (af-packet interface if I understood correctly, I got it with vppctl show int) ; MTU = 1500.
• VPP interface (tap0) ; MTU = 1450
• VPP interface (vxlan_tunnel0) ; MTU = 1450
• I have several veth (got with ip a in the host netns, but I haven't found how to identify the coresponding peers) ; MTU = 1500
• interface in the pod ; MTU = 1450

[edwarnicke]

@LionelJouin - we appear to have a fix.

I was able to reproduce your issue locally.

Tracked it back to: https://gerrit.fd.io/r/c/vpp/+/34585

Tested locally that it resolves your issue (it does!).

Incorporated in: edwarnicke/govpp#44
Which as soon as it gets in we can move into use in NSM in time for the 1.1 release :)

The net net issue was this:

		  if (tph->tp_snaplen > apif->host_mtu)
		    fill_gso_buffer_flags (first_b0, apif->host_mtu,
					   l4_hdr_sz);

in the af-packet-input node. It was mistakenly comparing the snaplen (which includes the ethernet header) to the host_mtu (which does not include the ethernet header) to determine whether the incoming packet was a GSO packet or not.

The vxlan packet was coming in via af-packet. The size of its 'payload' (where payload here is relative to the outer ethernet header) was 1494. Add the ethernet header (14 bytes) and the snaplen was 1508. 1508 was larger than the host-mtu (1500) and so it was being marked as GSO. When it got l2xc cross connected to the tap interface, which did not have GSO enabled... it was being dropped.

The fix correctly compensates for the outer ethernet header.

Some comments lest there be confusion:

• host_mtu is 1500 bytes
• mtu of the vxlan tunnel is 1450 (1500 -50 bytes for worst case vxlan overhead).

[edwarnicke]

@LionelJouin Once these these have been merged this should be ready for you to retest:

• Incorporate VPP GSO fix cmd-forwarder-vpp#411
• Incorporate VPP GSO fix cmd-nse-firewall-vpp#138
• Incorporate VPP GSO fix cmd-nse-icmp-responder-vpp#312
• Incorporate VPP GSO fix cmd-nsc-vpp#323
• Incorporate VPP GSO fix cmd-nse-vlan-vpp#7