After using the new version, there are errors with Intel's 82599 and E810 network cards, while Mellanox network cards are functioning normally. #321
Comments
|
Hi , @wangjun0728 .
Please, try the newer 22.11. There are numerous fixes in drivers between 22.11.1 and 22.11.4. |
|
Hi @igsilya ,I attempted to update DPDK to version 22.11.4, but the same error persists. E810： 82599: |
|
OK. I don't really know what could be wrong with an ice driver and I don't have any hardware to test with. The only suggestion here will be to try and update the firmware on the card in case you're not using the latest version. For the other driver we can try to debug that, but we need to know how these invalid packets look like. Also, what OVS version are you using? Maybe worth trying to update to the latest stable releases if you're not using them already. |
|
Hi, @igsilya thank you very much for your reply. The output log after I tried using your patch is as follows: Additionally, the OVS version I'm using is 2.17.5lts. However, now when I debug after merging the changes related to checksum and TSO, I encounter this issue. It was fine before the merge, and the main changes merged are as follows. However, it's not easy for me to fully upgrade OVS because I rely on the version of OVN. |
This should not be a problem. You should be able to upgrade OVS and OVN should still work just fine. The version of OVS you build OVN with and the one that you're using in runtime don't need to be the same. There is a build time dependency, because OVN is using some of the OVS libraries, but there is no runtime dependency because communication between OVS and OVN is happening over OpenFlow or OVSDB, which are stable protocols. Any version of OVN should be able to work with any version of OVS in runtime. So, you can build OVN with the version of OVS shipped in a submodule and use a separate newer version of OVS deployed on a host. Assuming you're using static linking, there should be no issues. In fact, that is a recommended way of using OVS with OVN. The checksum offloading patches had a lot of small issues, so I would not be surprised if some of the fixes got lost in backporting. I'll try to look at the dumps, but I'd still recommend you to just upgrade OVS on the node instead. |
So, these are Geneve packets and the offload is requested for the outer IPv4 checksum. Tunnel offloads were introduced in OVS 3.3, meaning they were not tested with DPDK older than 23.11. I would not be surprised that divers are missing some support or fixes. I don't think it makes sense to investigate this issue any further and I highly recommend you to just upgrade OVS and use it with supported version of DPDK. |
|
Hi @igsilya ，I do understand the usage scenario of geneve messages. Currently, the 82599 network card does not support offload the outer IP checksum and outer UDP checksum. Thank you very much for your suggestion. I will try the latest version of OVS 3.3 as soon as possible and provide a verification reply as soon as possible. Thank you again for your reply. |
|
Hi @igsilya ，I have completed the upgrade from OVS version to 3.3 and DPDK version to 23.11, but the same issue still exists。 E810： `2024-03-07T07:42:56.712Z|00341|dpdk|INFO|VHOST_CONFIG: (/var/run/openvswitch/vh-userclient-8d1fca5d-dc-vhostuser.sock) read message VHOST_USER_SET_VRING_ENABLE ovs-vsctl list open ovs-vsctl get interface tun_port_p0 status 82599： `2024-03-07T07:46:37.430Z|00002|netdev_dpdk(pmd-c02/id:88)|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported ovs-vsctl list open ovs-vsctl get interface tun_port_p0 status |
|
Regarding E810, it was observed that there was an abnormal printing message after I created the vhost user client port. |
|
@wangjun0728 thanks for the info! CC: @mkp-rh |
|
@igsilya Additionally, the latest code I'm using is this one：https://github.com/openvswitch/ovs/commits/branch-3.3/ The checksum offload capability of Intel network cards indeed differs from Mellanox network cards. I believe this might be the root cause of the issue, as it seems more like a problem with the DPDK-side driver. |
|
I think there's somewhat of a hint provided here:
There are very few places where DPDK will return ENOTSUPP. I don't have an E810 card right now, but will try to investigate the code. |
|
@mkp-rh note that |
|
For the MDD issue, I see that the E810 errata page reports:
So it could be the MDD anti-spoofing features, or a general tx data check failure. In the ixgbe driver, ixgbe_prep_pkts only returns ENOTSUP if the ol_flags are incorrect. From the log above I see ol_flags=0x800800000000182, which when translates into the following tx offload flags:
ixgbe_rxtx.c contains the supported IXGBE_TX_OFFLOAD_MASK, which doesn't include RTE_MBUF_F_TX_TUNNEL_GENEVE. So that flag shouldn't be included when we send the frame. |
So, if we do not request TSO or inner checksumming we must not specify |
|
@mkp-rh Hmm, also the Edit: Nevermind, wrong flag. But the existing check might still be incomplete. |
|
@wangjun0728 Could you try this one: igsilya/ovs@00c0a91 ? It should fix the 82599 case at least, I think. |
|
@igsilya This looks great! Applying your modifications resolved the error with the 82599 network card, and I can now communicate without issues using iperf. Additionally, I've observed the E810 network card, and the MDD error still persists. |
|
I also noticed a modification in the DPDK community, but applying it didn't yield any results. I suspect there might be a flaw in the E810 driver's support for tunnel TSO. https://patches.dpdk.org/project/dpdk/patch/[email protected]/ |
|
After enabling DPDK's PMD logs with the command --log-level=pmd,debug, I captured a portion of DPDK startup log information. Currently, it's unclear whether there's any definite correlation with the errors present. 2024-03-11T02:38:04.088Z|00007|dpdk|INFO|Using DPDK 23.11.0 |
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For exmaple, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks geneve tunnels on these cards. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Ilya Maximets <[email protected]>
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For exmaple, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks geneve tunnels on these cards. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Ilya Maximets <[email protected]>
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For exmaple, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks geneve tunnels on these cards. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Ilya Maximets <[email protected]>
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For exmaple, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks geneve tunnels on these cards. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Ilya Maximets <[email protected]>
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For example, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks Geneve tunnels on these cards. An extra check is added to make sure we don't have any unexpected Tx offload flags set. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Ilya Maximets <[email protected]> Signed-off-by: 0-day Robot <[email protected]>
|
@wangjun0728 I posted the refined verion of the 82599 fix here: https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ |
|
For the E810, I still don't have a lot to suggest. One thing that might help understanding the situation better is to dump some of the mbufs we're trying to send. Maybe you can capture some logs with the following change applied: ? |
|
@wangjun0728 Are you able to check if the following patch resolves your issue on E810? |
|
Rolling back your modification didn't resolve the issue; it seems that the 82599 network card doesn't support enabling TSO. The same issue exists on the Mellanox CX5 network card. Even though the MTU has been adjusted to 1558, iperf cannot send a large number of TCP packets. However, unlike the 82599 network card, no similar errors have been observed with the Mellanox CX5 card. So, does it mean that TSO cannot be enabled if the outer UDP checksum offload is not supported? |
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Signed-off-by: 0-day Robot <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Signed-off-by: 0-day Robot <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: 0-day Robot <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Signed-off-by: 0-day Robot <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Ilya Maximets <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Ilya Maximets <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
Some drivers (primarily, Intel ones) do not expect any marking flags being set if no offloads are requested. If these flags are present, driver will fail Tx preparation or behave abnormally. For example, ixgbe driver will refuse to process the packet with only RTE_MBUF_F_TX_TUNNEL_GENEVE and RTE_MBUF_F_TX_OUTER_IPV4 set. This pretty much breaks Geneve tunnels on these cards. An extra check is added to make sure we don't have any unexpected Tx offload flags set. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Acked-by: Mike Pattrick <[email protected]> Signed-off-by: Ilya Maximets <[email protected]>
Fixing the issue of incorrect outer UDP checksum in packets sent by E810 or X710. We disable RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM,but also disable all the dependent offloads like RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO and RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO. Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Reported-at: openvswitch/ovs-issues#321 Signed-off-by: Jun Wang <[email protected]> Signed-off-by: Ilya Maximets <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]>
The outer checksum offloading API in DPDK is ambiguous and was implemented by Intel folks in their drivers with the assumption that any outer offloading always goes with an inner offloading request. With net/i40e and net/ice drivers, in the case of encapsulating a ARP packet in a vxlan tunnel (which results in requesting outer ip checksum with a tunnel context but no inner offloading request), a Tx failure is triggered, associated with a port MDD event. 2024-03-27T16:02:07.084Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event To avoid this situation, if no checksum or segmentation offloading is requested on the inner part of a packet, fallback to "normal" (non outer) offloading request. Reported-at: openvswitch/ovs-issues#321 Fixes: 084c808 ("userspace: Support VXLAN and GENEVE TSO.") Fixes: f81d782 ("netdev-native-tnl: Mark all vxlan/geneve packets as tunneled.") Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]> (cherry picked from commit 2e03f55) Signed-off-by: Roi Dayan <[email protected]> Change-Id: Ibc94d237c35d785aed8921e9e5c6cac29dbd7ea7
In a typical setup like: guest A <-virtio-> OVS A <-vxlan-> OVS B <-virtio-> guest B TSO packets from guest A are segmented against the OVS A physical port mtu adjusted by the vxlan tunnel header size, regardless of guest A interface mtu. As an example, let's say guest A and guest B mtu are set to 1500 bytes. OVS A and OVS B physical ports mtu are set to 1600 bytes. Guest A will request TCP segmentation for 1448 bytes segments. On the other hand, OVS A will request 1498 bytes segments to the HW. This results in OVS B dropping packets because decapsulated packets are larger than the vhost-user port (serving guest B) mtu. 2024-04-17T14:13:01.239Z|00002|netdev_dpdk(pmd-c03/id:7)|WARN|vhost0: Too big size 1564 max_packet_len 1518 vhost-user ports expose a guest mtu by filling mbuf->tso_segsz. Use it as a hint. This may result in segments (on the wire) slightly shorter than the optimal size. Reported-at: openvswitch/ovs-issues#321 Signed-off-by: David Marchand <[email protected]> Acked-by: Kevin Traynor <[email protected]> Signed-off-by: Kevin Traynor <[email protected]> (cherry picked from commit a924887) Signed-off-by: Roi Dayan <[email protected]> Change-Id: Iaeb59ea36b3fd22b8befc753abf03f5a0f5fee42
|
I use this patch from Mike, which can very well solve the problem that my ov-dpdk cannot enable TSO. Currently, I have verified that CX6/X710/E810, etc. are all normal. The performance of virtual machines on the same node is very high, and cross-node performance is still good. https://patchwork.ozlabs.org/project/openvswitch/list/?series=417313 |
The DPDK version is 22.11. Currently, it appears that the DPDK errors are occurring due to the new version's checksum offload. Mellanox network cards seem to be operating normally. However, both E810 and 82599 network cards are displaying different error messages.
E810：
{bus_info="bus_name=pci, vendor_id=8086, device_id=159b", driver_name=net_ice, if_descr="DPDK 22.11.1 net_ice", if_type="6", link_speed="25Gbps", max_hash_mac_addrs="0", max_mac_addrs="64", max_rx_pktlen="1618", max_rx_queues="256", max_tx_queues="256", max_vfs="0", max_vmdq_pools="0", min_rx_bufsize="1024", n_rxq="2", n_txq="5", numa_id="1", port_no="1", rx-steering=rss, rx_csum_offload="true", tx_geneve_tso_offload="false", tx_ip_csum_offload="true", tx_out_ip_csum_offload="true", tx_out_udp_csum_offload="true", tx_sctp_csum_offload="true", tx_tcp_csum_offload="true", tx_tcp_seg_offload="false", tx_udp_csum_offload="true", tx_vxlan_tso_offload="false"}
error:
2024-03-04T10:57:01.102Z|00018|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
2024-03-04T10:57:01.105Z|00019|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
2024-03-04T10:57:01.113Z|00020|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
2024-03-04T10:57:01.167Z|00021|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
2024-03-04T10:57:01.278Z|00022|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
2024-03-04T10:57:01.599Z|00023|dpdk|WARN|ice_interrupt_handler(): OICR: MDD event
82599：
{bus_info="bus_name=pci, vendor_id=8086, device_id=10fb", driver_name=net_ixgbe, if_descr="DPDK 22.11.1 net_ixgbe", if_type="6", link_speed="10Gbps", max_hash_mac_addrs="4096", max_mac_addrs="127", max_rx_pktlen="1618", max_rx_queues="128", max_tx_queues="64", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1024", n_rxq="2", n_txq="5", numa_id="0", port_no="1", rx-steering=rss, rx_csum_offload="true", tx_geneve_tso_offload="false", tx_ip_csum_offload="true", tx_out_ip_csum_offload="false", tx_out_udp_csum_offload="false", tx_sctp_csum_offload="true", tx_tcp_csum_offload="true", tx_tcp_seg_offload="false", tx_udp_csum_offload="true", tx_vxlan_tso_offload="false"
error:
2024-03-04T11:04:52.740Z|00384|netdev_dpdk|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported
2024-03-04T11:04:54.449Z|00385|netdev_dpdk|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported
2024-03-04T11:04:55.492Z|00386|netdev_dpdk|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported
2024-03-04T11:04:55.592Z|00387|netdev_dpdk|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported
2024-03-04T11:04:56.644Z|00388|netdev_dpdk|WARN|tun_port_p1: Output batch contains invalid packets. Only 0/1 are valid: Operation not supported
mellanox：
{bus_info="bus_name=pci, vendor_id=15b3, device_id=1017", driver_name=mlx5_pci, if_descr="DPDK 22.11.1 mlx5_pci", if_type="6", link_speed="25Gbps", max_hash_mac_addrs="0", max_mac_addrs="128", max_rx_pktlen="1618", max_rx_queues="1024", max_tx_queues="1024", max_vfs="0", max_vmdq_pools="0", min_rx_bufsize="32", n_rxq="2", n_txq="5", numa_id="3", port_no="1", rx-steering=rss, rx_csum_offload="true", tx_geneve_tso_offload="false", tx_ip_csum_offload="true", tx_out_ip_csum_offload="true", tx_out_udp_csum_offload="false", tx_sctp_csum_offload="false", tx_tcp_csum_offload="true", tx_tcp_seg_offload="false", tx_udp_csum_offload="true", tx_vxlan_tso_offload="false"}
The text was updated successfully, but these errors were encountered: