Mutiple physical interface support for secondary network bridge

[john5985]

Describe the problem/challenge you have

When I used secondary network bridge feature, I specified more than one port in secondary bridge, but only one show up:
secondaryNetwork:
ovsBridges: [{"bridgeName": "br-kbear", "physicalInterfaces": ["enp1s0f1v0", "enp1s0f1v1"]}]

Given most SRIOV NICs have 8 HW queues, my request is to support up to 8 ports.

Describe the solution you'd like

secondaryNetwork:
ovsBridges: [{"bridgeName": "br-kbear", "physicalInterfaces": ["enp1s0f1v0", "enp1s0f1v1", "enp1s0f1v2", "enp1s0f1v3", "enp1s0f1v4", "enp1s0f1v5", "enp1s0f1v6", "enp1s0f1v7"]}]

Anything else you would like to add?

[aroradaman]

/assign

[jianjuns]

@john5985 : in your case, are all these interfaces connect to the same underlay subnet/L2? If yes, do you assume we auto create a bound for these interfaces when connecting to the OVS bridge?

[john5985]

@jianjuns Thank you for following up! I would not group all these interfaces to the same underlay; each interface is associated with a L2 VLAN segment and a private subnet over the OVS bridge. I followed your example of VLAN based network attachment and configured this VLAN as a router with SRIOV interface as WAN.

[jianjuns]

But in this case, do you assume OVS will add the VLAN tag for the traffic? That means we should include a VLAN ID parameter on the physical interface setting in the Antrea secondary OVS bridge configuration.

[john5985]

I assume OVS will implicitly push vlan tag when traffic entering OVS and pop vlan tag when traffic leaving from this port.
I see -- yes I manually set vlan tag to this sriov port using ovs-vsctl command.

[john5985]

I think you could default sriov port to trunk which means no vlan, or one or multiple vlan tags:
trunk: []. # do nothing
trunk: [10] # ovs-vsctl set port myport tag=10, which is my configuration
trunk: [10,20] # ovs-vsctl set port myport trunks=10,20

[jianjuns]

Humm.. Have you checked if no VLAN ID on the OVS port, will OVS still insert the VLAN tag before sending the packets out?

Also broadcast from one Pod will be sent to all physical interfaces. Maybe it is ok as physical network should drop the packets if the VLAN does not match.

@aroradaman is working on the implementation. He may comment with his test results.

[john5985]

Broadcast is exactly what I am worried about. The traffic from SRIOV interfaces may come from the same (set of) gateways; I didn't want ARP traffic broadcast between these sriov interfaces thus using vlan to isolate.

[aroradaman]

Humm.. Have you checked if no VLAN ID on the OVS port, will OVS still insert the VLAN tag before sending the packets out?

If I simply attach multiple physical ports with no VLAN ID to the secondary bridge then my kind of cluster networking gets affected and I lose connection to the API Server.

However, If I manually create a bond of all physical interfaces everything is fine and the reachability test passes.

[john5985]

In my case, I made one of physical port as trunk (no tag) while all other tagged with one particular VLAN; they are all set with ovs-vsctl set port tag

[aroradaman]

In my case, I made one of physical port as trunk (no tag) while all other tagged with one particular VLAN; they are all set with ovs-vsctl set port tag

I tried to replicate the same thing in the kind cluster, reachability/ping tests were working fine.
(infra setup: aroradaman@3b138ee)