- Testcases status
- Testbed
- Instruction to run MACsec test
- Common Configuration
- Test cases
- Testcase : Macsec Functionality
- Testcase : Macsec feature interop with other protocols
- Testcase : Deployment usecases and fault handling scenario's
- Link flap on an interface with macsec configured
- Link flap on a portchannel member which has macsec configured
- Testcases: Operation portchannel remove and re-add members to fix CRC error
- MACsec session cannot be established under wrong MKA configuration
- Config reload done on DUT with macsec configuration
- Everflow, port mirroring on macsec enabled interfaces
- Testcase : Macsec scenario's for multi-asic, multi-dut
- Testcase : Scale tests
| Test cases | status | comment |
|---|---|---|
| Check Control plane | Completed | |
| Check the Data plane | Completed | |
| Refresh SAK - timer based | Completed | |
| Refresh SAK - packet number based | Ongoing | |
| MACsec Key rotation, Primary/Fallback CAK | Not start | Feature hasn't been supported |
| MACsec Counters | Ongoing | |
| COPP | Ongoing | |
| Port channel with MACsec | Completed | |
| LLDP with MACsec | Completed | |
| BGP with MACsec | Completed | |
| PFC with MACsec | Not start | Feature hasn't been supported |
| SNMP with MACsec | Completed | |
| Link flap on an interface with MACsec | Completed | |
| Link flap on a PortChannel member with MACsec | Completed | |
| Operation PortChannel remove and re-add members to fix CRC error | Removed | No use case |
| MACsec session cannot be established under wrong MKA configuration | Completed | |
| Config reload done on DUT with macsec configuration | Completed | |
| Everflow, port mirroring on macsec enabled interfaces | Removed | Directly leveraging the testcase in everflow part |
| MACsec scenario's for multi-asic, multi-dut | Ongoing | |
| Large number of interfaces having macsec enabled on the DUT/linecard | Ongoing | |
| MACsec enabled on all interfaces and the DUT is rebooted | Ongoing |
+-------------------------------------------------------------------------+
| |
| DUT |
| |
+-------+-------------------+-----------------+--------------+----------+-+
* * | | |
* * | | |
*......... *......... +...... +...... |
* : * : | : | : |
* : * : | : | : |
+-----+------+ : +-----+------+ : +---+---+ : +---+---+ : |
|VM0 | : |VM1 | : |VM2 | : |VM3 | : |
|(Controlled)| : |(Controlled)| : | | : | | : |
+------------+ : +------------+ : +-------+ : +-------+ : |
: : : : |
+----------------+-------------------+--------------+--------------+----+-+
| |
| PTF |
| |
+-------------------------------------------------------------------------+
----- normal link
..... injected link
***** protected link
VM<->DUT up link
PTF<->DUT down linkIn this topology, We pick two VMs (MACsec support) that act as the MACsec participants of the DUT. These two pairs of MACsec participant belong to different MACsec connectivity association(CA).
About the detail to set the testbed, please refer the doc: https://github.com/sonic-net/sonic-mgmt/blob/master/docs/testbed/README.testbed.VsSetup.md firstly.
./testbed-cli.sh -m veos_vtb -n 4 -k vsonic start-vms server_1 password.txt
./testbed-cli.sh -t vtestbed.csv -m veos_vtb -k vsonic add-topo vms-kvm-t0 password.txt
./testbed-cli.sh -t vtestbed.csv -m veos_vtb deploy-mg vms-kvm-t0 veos_vtb password.txt./run_tests.sh -u -n vms-kvm-t0 -d vlab-01 -c macsec/test_macsec.py -f vtestbed.csv -i veos_vtb -e "--neighbor_type=sonic"MACsec profile table
| Field | Value |
|---|---|
| priority | DUT(64) VM0(63) VM1(65) |
| cipher suite | GCM-AES-128/GCM-AES-256/GCM-AES-XPN-128/GCM-AES-XPN-256 |
| CKN | 6162636465666768696A6B6C6D6E6F707172737475767778797A303132333435 |
| CAK | GCM-AES-128(0123456789ABCDEF0123456789ABCDEF)/GCM-AES-256(0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF) |
| policy | security |
| enable_replay_protect | true/false |
| replay_window | enable_replay_protect(0)/disable_replay_protect(100) |
| send_sci | true/false |
| rekey_period | 0/60 |
Port table
| Field | Value |
|---|---|
| pfc_encryption_mode | bypass/encrypt/strict_encrypt |
NOTE: The macsec functionality testcases will cover the various options for each of the above supported fields
This testcase covers the macsec/MKA protocol functionality
-
Enable macsec on the interface on the DUT and remote VM.
-
Check the process,
wpa_supplicant, for the target port is running in the devices. -
Check APP DB, Check the following fields in MACsec port table are consistent with configuration
Config DB Field Config DB Value App DB Field APP DB Value enable true cipher_suite GCM-AES-128 cipher_suite GCM-AES-128 cipher_suite GCM-AES-256 cipher_suite GCM-AES-256 cipher_suite GCM-AES-XPN-128 cipher_suite GCM-AES-XPN-128 cipher_suite GCM-AES-XPN-256 cipher_suite GCM-AES-XPN-256 enable_protect true policy security enable_encrypt true policy integrity_only enable_encrypt false send_sci true send_sci true send_sci false send_sci false -
Check the following fields in MACsec SC table and MACsec SA table are consistent
- There should be a MACsec SA in MACsec SA table with the same AN of encoding_an in MACsec SC.
- The count of ingress MACsec SA shouldn't be lesser than the count of egress MACsec SA in the peer side.
- The corresponding ingress and egress MACsec SA should have same sak and auth_key.
- The next_pn of egress MACsec SA shouldn't be lesser than the lowest_acceptable_pn of the corresponding ingress MACsec SA in the peer side.
-
Check MKA session
- Get the MKA session
- In the virtual SONiC switch:
ip macsec show - In the physical SONiC switch:
show macsec(SONiC CLI)
- In the virtual SONiC switch:
- Check the MACsec session is consistent with configuration.
- Get the MKA session
+-----------------------------------------------------------------------------------+
| |
| DUT |
| |
+-------+-------------------+-------------------+-------------------+-------------+-+
* * | | |
* * | | |
*......... *......... +......... +......... |
* : * : | : | : |
* : * : | : | : |
+-----+------+ : +-----+------+ : +-----+------+ : +-----+------+ : |
|VM0 | : |VM1 | : |VM2 | : |VM3 | : |
|(Controlled)| : |(Controlled)| : | | : | | : |
|ptf_nn_agent| : |ptf_nn_agent| : |ptf_nn_agent| : |ptf_nn_agent| : |
+------------+ : +------------+ : +------------+ : +------------+ : |
: : : : |
+----------------+-------------------+-------------------+-------------------+----+-+
| |
| PTF (ptf_nn_agent) |
| |
+-----------------------------------------------------------------------------------+
----- normal link
..... injected link
***** protected link
VM<->DUT up link
PTF<->DUT down linkAll VMs and PTF docker in the host need to install PTF NN agent. So, SONiC-mgmt-docker can use an unified interface, ptf_nn_client, to handle the packets sending operation in the servers and VMs.
-
Check PTF to VM traffic. Verify the traffic is truly protected by MACsec.
- Send IPv4 packet
Field Value ether dst DUT mac address ip src 1.2.3.4 ip dst VM ipv4 address ip ttl 64 - Expected IPv4 packet
Field Value ether src DUT mac address ether dst VM mac address ip src 1.2.3.4 ip dst VM ipv4 address ip ttl 63 - Send a set of above packet on the down link of DUT
- The target VM should receive at least one expected above packet
- In the injected port of PTF, we should get at least one expected packet encapsulated by MACsec
Notes 1. The number of send packet is 100 to avoid the send packet dropped by MACsec engine or others. 2. Set the buffer queue of PTF to the 1000 to avoid the send packet dropped by PTF 3. We can decapsulate all MACsec packets by the activated SAK in the APP DB. Because the operation of decapsulation needs to take a long time which may cause we miss the expected packet, we collect all packets for 10 seconds firstly and decapsulate them one by one until the expected packet appearance.
-
Check VM to VM traffic, This test is to verify a packet can be correctly forwarded between controlled nodes and uncontrolled nodes. In the following statement, we assume we send packet from VM(0) to VM(1). But in the real test, we will test all directions from VM(x) to VM(non-x)
- Send IPv4 packet
Field Value ether dst DUT mac address ip src VM0 ipv4 address ip dst VM1 ipv4 address ip ttl 64 - Expected IPv4 packet
Field Value ether dst DUT mac address ether dst DUT mac address ip src VM0 ipv4 address ip dst VM1 ipv4 address ip ttl 63 - Send a set of above packet on the VM0
- VM1 should receive at least one expected above packet
- Check the interface stats with macsec counters.
The thresholds of rekey packet number are 0xC0000000ULL to 32bits packet number and 0xC000000000000000ULL to 64bits packet number(XPN). It's impossible to really send many packets to trigger the rekey action. So, We use the attribute next_pn of MACSEC_EGRESS_SA in APP_DB to cheat MKA protocol for rekey action.
┌──────────────┐
│ │
│ sonic-mgmt │
│ │
└────┬─────────┘
│
next_pn=threshold-100
│ ┌───────────────────────────┐
│ │ │
│ │ wpa_supplicant │
│ │ │
│ └─────────────────────────▲─┘
│ * │
│ * │
│ rekey transmit_next_pn
│ * │
│ * │
┌───▼────────▼─┐ ┌──────────────┴─┐
│ │ │ │
│ app_db │ │ counter_db │
│ │ │ │
└───┬──────────┘ └──────────────▲─┘
│ │
│ │
│ │
MACSEC_EGRESS_SA:next_pn │
│ │
┌───▼──────────┐ │
│ │ │
│ orchagent │ SAI_MACSEC_SA_ATTR_CURRENT_XPN
│ │ │
└───┬──────────┘ │
│ │
SAI_MACSEC_SA_ATTR_CONFIGURED_EGRESS_XPN │
│ │
│ │
┌───▼──────────────────────────────┴──┐
│ │
│ syncd │
│ │
└─────────────────────────────────────┘-
Steps
- Record the SAK in APP DB.
- Update the next_pn of egress SA to
threshold - 100. - Ping VM0
sudo ping VM0_ipv4_address -c 200 -i 0.1to simulate continuous traffic. - Check whether the SAK was changed. If no, sleep 6 seconds and check again until waiting more 10 times(60 seconds) and this test fail. If yes, this test pass.
- Check whether the new AN is next expected AN.
- Expect no packet loss on the ping result.
-
Periodic Rekey, this testcase is only available if the field rekey_period in configuration is more than 0.
- Record the SAK in APP DB.
- Ping VM0
sudo ping VM0_ipv4_address -w 120 -q -i 0.1to simulate continuous traffic. - Check whether the SAK was changed.
- Check whether the new AN is next expected AN.
- Expect no packet loss on the ping result.
TODO
MACsec counter helps the monitoring of MACsec frame TX/RX on protected (macsec-enabled) link. This test is to verify MACsec statistics can be collected from ASIC/NPU/PHY and it includes proper counters to indicate the status of MACsec frame handling.
- Check MACsec SA counters
-
Send traffic continuously by referring to the test Check the Data plane
-
For egress SA, the following counters keep increasing:
- SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED
- SAI_MACSEC_SA_STAT_OCTETS_PROTECTED
- SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED
- SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED
-
For ingress SA, the following counters keep increasing:
- SAI_MACSEC_SA_STAT_IN_PKTS_OK
- SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED
- SAI_MACSEC_SA_STAT_OCTETS_PROTECTED
-
For both egress and ingress SAs, the following counter keeps increasing:
- SAI_MACSEC_SA_ATTR_CURRENT_XPN
-
TODO
This testcase covers the behavior of slow protocols when mac security is configured on interfaces
Note: Below test expectations are based on assumption that physical interface remains up when macsec profile is attached to an interface.
-
Configure the macsec profile and apply them on a selected interface. Let the MKA session be establised
- Add this macsec enabled interface as member of a Portchannel
- Expect the member interface and portchannel to be in oper UP state.
- Add this macsec enabled interface as member of a Portchannel
-
Configure macsec on the member interface of a Portchannel which is already in oper UP state. There is only one member interface.
- Expect the portchannel to remain oper UP if the mka session establishment happens within 3*30sec, assuming LACP is in slow mode.
- Expect the portchannel to go down if time taken for mka session establishment is > 3*30sec.
- Portchannel interface goes oper UP after the MKA session is established
- Configure the macsec profile on interface where LLDP neighborship was already present
- Expect the LLDP neighborship is maintained as long as mka session establishment happens within 4*30sec which is default LLDP hold time interval
- Remove the macsec profile from the interface
- Check the LLDP neighborship exists even after the removal of macsec config.
- Check the behaviour when macsec is enabled on an interface where BGP session was already established with peer.
- Expect to see BGP neighbors remain established state as long as mka session establishment happens within the BGP hold time interval
- Remove the macsec profile from the interface
- Check the BGP sessions are established again after removal of macsec config.
Use PTF to generate and capture PFC packets and set the same mode between DUT and Neighbor device.
The switch should react clear and encrypted ingress PFC frames and should send clear egress PFC frames.
- Send clear PFC frame from the neighbor device to the DUT
- The DUT expects to capture the clear PFC packet
- Send clear PFC frame from the DUT to the neighbor device
- The neighbor expect to capture the clear PFC packet
- The inject port expects to capture the clear PFC packet
- Send clear PFC frame on the PTF injected port
- The DUT expects to capture the clear PFC packet
- Send encrypted PFC frame on the PTF injected port
- The DUT expects to capture the clear PFC packet
The switch should react clear and encrypted PFC frames, send encrypted PFC frames.
- Send clear PFC frame from the neighbor device to the DUT
- The DUT expects to capture the clear PFC packet
- Send clear PFC frame from the DUT to the neighbor device
- The neighbor expect to capture the clear PFC packet
- The inject port expects to capture the encrypted PFC packet
- Send clear PFC frame on the PTF injected port
- The DUT expects to capture the clear PFC packet
- Send encrypted PFC frame on the PTF injected port
- The DUT expects to capture the clear PFC packet
The switch should only react encrypted PFC frames, send encrypted PFC frames.
- Send clear PFC frame from the neighbor device to the DUT
- The DUT expects to capture the clear PFC packet
- Send clear PFC frame from the DUT to the neighbor device
- The neighbor expect to capture the clear PFC packet
- The inject port expects to capture the encrypted PFC packet
- Send clear PFC frame on the PTF injected port
- The DUT expects to drop any PFC packet
- Send encrypted PFC frame on the PTF injected port
- The DUT expects to capture the clear PFC packet
- Configure the macsec profile on interface and check if the snmp walk succeeds from the peer VM.
This testcase covers the various fault scenario's and the expected behavior. The link flap will happen in both local and remote interface down/up.
- MKA session can be recovered from the link flap if the port comes back up in < 6 secs (MKA lietime)
- If the port is down for more than 6 sec, MKA session will create a new session.
- When the member interface flaps and it is the only portchannel member
- Expect the Portchannel to go down and come up depending on whether the member port comes back in 6 secs (MKA lietime)
- When one member interface flaps, but the Portchannel has more member ports which are macsec enabled.
- Expect Portchannel to remain up.
The portchannel member is removed from the portchannel, add/remove IP address. Add the interface back to portchannel with macsec enabled. Check the behavior back
- If the CAK is mis-matched between DUT and peer, the MKA session cannot be established.
- Control plane protocols eg: BGP session will not be established.
- The macsec sessions will be reconfigured, MKA session created again with a new SAK key.
- The control protocol sessions like LACP, LLDP, BGP get established again over the macsec configured interfaces.
- This test is to verify 2 cases
- configure mirroring on interfaces where macsec is configured.
- configure macsec on the outgoing interface ( in case of Everflow ).
- TODO add expected behavior
- Verify that macsec dockers are coming up in different namespaces.
- Verify macsec packet flow where the Ingress and Egress ports are on different Linecards.
- TODO add expected behavior
- Use the "configure feature macsec enabled" command to turn ON macsec, with all ports having macsec profile attached.
- Check the CPU, ASIC behavior when there are multiple wpa_supplicant processes being spawned.
- Expect the macsec sessions all come up. Measure the time taken.
- When all the interfaces flap together, how much time it takes for Portchannels/BGP sessions to be up -Expect the macsec sessions all come up. Measure the time taken.
- Enable macsec on multiple interfaces so that they have same rekey period.
- Check the sessions are coming back up and there is no traffic loss during h/w progamming.
- Check the macsec docker comes up and macsec sessions are established.