[dualtor] Add script to verify consistency between kernel and ASIC #2840

lolyu · 2023-05-17T13:30:06Z

Work item tracking

Microsoft ADO (number only): 22571694

What I did

Add script dualtor_neighbor_check.py to verify the neighbor consistency
based on the mux state. It will have the following output:

NEIGHBOR       MAC                PORT        MUX_STATE    IN_MUX_TOGGLE    NEIGHBOR_IN_ASIC    TUNNERL_IN_ASIC    HWSTATUS
-------------  -----------------  ----------  -----------  ---------------  ------------------  -----------------  ----------
192.168.0.2    ee:86:d8:46:7d:01  Ethernet4   standby      False            no                  yes                consistent
192.168.0.3    86:73:c2:22:bf:02  Ethernet8   standby      False            no                  yes                consistent
192.168.0.24   56:a6:bf:c5:dd:17  Ethernet92  active       False            yes                 no                 consistent
192.168.0.25   3a:18:56:f5:02:18  Ethernet96  active       False            yes                 no                 consistent
192.168.0.100  00:00:00:00:00:00  N/A         N/A          N/A              no                  yes                consistent

Signed-off-by: Longxiang Lyu [email protected]

How I did it

the workflow of this scripts:

for non-zero-mac neighbors in APPL_DB NEIGH_TABLE, use the ASIC_DB fdb entries to find the mux port that it belongs to.
check if the neighbor is consistent with mux state:
- if mux state is active, the neighbor is consistent only if the neighbor is present in ASIC_DB but no tunnel route.
- if mux state is standby, the neighbor is consistent only if the tunnel route is present in ASIC_DB1 but no neighbor.
if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs.

How to verify it

UT and verify on testbed.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

scripts/dualtor_neighbor_check.py

yxieca · 2023-05-24T22:45:12Z

@lolyu Are you able to pass the coverage checker?

bingwang-ms · 2023-05-24T22:50:08Z

Discussed with Guohan offline, is it better if we integrate this script with existing route_checker.py? Both scripts are for consistency checking.

lolyu · 2023-05-25T02:15:18Z

Discussed with Guohan offline, is it better if we integrate this script with existing route_checker.py? Both scripts are for consistency checking.

It is hard to integrate this with route_checker.py.
First, dualtor_neighbor_check.py has more tables to read, and it is based on a Lua script to read the db, and route_check.py uses the swsscommon table subscriptions to read table contents and its updates, its two different design.
Second, the mux states changes much frequently than the routes, especially in the mux flapping scenario, route_check.py reads the tables with swsscommon table subscriptions, so it is hard to capture a snapshot of the database and get accurate check result.

Signed-off-by: Longxiang Lyu <[email protected]>

What I did As the subject. This PR is to leave the vlan neighbor route checking to dualtor_neighbor_check.py script: #2840 Signed-off-by: Longxiang Lyu [email protected] How I did it If any misses are found on dualtor, ignore those vlan neighbor misses. How to verify it UT and verify on testbed.

scripts/dualtor_neighbor_check.py

StormLiangMS · 2023-07-19T08:32:11Z

@lolyu do we have ADO for this? Pls add tag for branches which need this fix.

What I did As the subject. This PR is to leave the vlan neighbor route checking to dualtor_neighbor_check.py script: #2840 Signed-off-by: Longxiang Lyu [email protected] How I did it If any misses are found on dualtor, ignore those vlan neighbor misses. How to verify it UT and verify on testbed.

…2840) #### What I did Add script `dualtor_neighbor_check.py` to verify the neighbor consistency based on the mux state. It will have the following output: ``` NEIGHBOR MAC PORT MUX_STATE IN_MUX_TOGGLE NEIGHBOR_IN_ASIC TUNNERL_IN_ASIC HWSTATUS ------------- ----------------- ---------- ----------- --------------- ------------------ ----------------- ---------- 192.168.0.2 ee:86:d8:46:7d:01 Ethernet4 standby False no yes consistent 192.168.0.3 86:73:c2:22:bf:02 Ethernet8 standby False no yes consistent 192.168.0.24 56:a6:bf:c5:dd:17 Ethernet92 active False yes no consistent 192.168.0.25 3a:18:56:f5:02:18 Ethernet96 active False yes no consistent 192.168.0.100 00:00:00:00:00:00 N/A N/A N/A no yes consistent ``` Signed-off-by: Longxiang Lyu <[email protected]> #### How I did it the workflow of this scripts: 1. for non-zero-mac neighbors in `APPL_DB` `NEIGH_TABLE`, use the `ASIC_DB` fdb entries to find the mux port that it belongs to. 2. check if the neighbor is consistent with mux state: * if mux state is `active`, the neighbor is consistent only if the neighbor is present in `ASIC_DB` but no tunnel route. * if mux state is `standby`, the neighbor is consistent only if the tunnel route is present in `ASIC_DB`1 but no neighbor. 3. if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs. #### How to verify it UT and verify on testbed. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

What I did As the subject. This PR is to leave the vlan neighbor route checking to dualtor_neighbor_check.py script: sonic-net#2840 Signed-off-by: Longxiang Lyu [email protected] How I did it If any misses are found on dualtor, ignore those vlan neighbor misses. How to verify it UT and verify on testbed.

…onic-net#2840) #### What I did Add script `dualtor_neighbor_check.py` to verify the neighbor consistency based on the mux state. It will have the following output: ``` NEIGHBOR MAC PORT MUX_STATE IN_MUX_TOGGLE NEIGHBOR_IN_ASIC TUNNERL_IN_ASIC HWSTATUS ------------- ----------------- ---------- ----------- --------------- ------------------ ----------------- ---------- 192.168.0.2 ee:86:d8:46:7d:01 Ethernet4 standby False no yes consistent 192.168.0.3 86:73:c2:22:bf:02 Ethernet8 standby False no yes consistent 192.168.0.24 56:a6:bf:c5:dd:17 Ethernet92 active False yes no consistent 192.168.0.25 3a:18:56:f5:02:18 Ethernet96 active False yes no consistent 192.168.0.100 00:00:00:00:00:00 N/A N/A N/A no yes consistent ``` Signed-off-by: Longxiang Lyu <[email protected]> #### How I did it the workflow of this scripts: 1. for non-zero-mac neighbors in `APPL_DB` `NEIGH_TABLE`, use the `ASIC_DB` fdb entries to find the mux port that it belongs to. 2. check if the neighbor is consistent with mux state: * if mux state is `active`, the neighbor is consistent only if the neighbor is present in `ASIC_DB` but no tunnel route. * if mux state is `standby`, the neighbor is consistent only if the tunnel route is present in `ASIC_DB`1 but no neighbor. 3. if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs. #### How to verify it UT and verify on testbed. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

…2840) #### What I did Add script `dualtor_neighbor_check.py` to verify the neighbor consistency based on the mux state. It will have the following output: ``` NEIGHBOR MAC PORT MUX_STATE IN_MUX_TOGGLE NEIGHBOR_IN_ASIC TUNNERL_IN_ASIC HWSTATUS ------------- ----------------- ---------- ----------- --------------- ------------------ ----------------- ---------- 192.168.0.2 ee:86:d8:46:7d:01 Ethernet4 standby False no yes consistent 192.168.0.3 86:73:c2:22:bf:02 Ethernet8 standby False no yes consistent 192.168.0.24 56:a6:bf:c5:dd:17 Ethernet92 active False yes no consistent 192.168.0.25 3a:18:56:f5:02:18 Ethernet96 active False yes no consistent 192.168.0.100 00:00:00:00:00:00 N/A N/A N/A no yes consistent ``` Signed-off-by: Longxiang Lyu <[email protected]> #### How I did it the workflow of this scripts: 1. for non-zero-mac neighbors in `APPL_DB` `NEIGH_TABLE`, use the `ASIC_DB` fdb entries to find the mux port that it belongs to. 2. check if the neighbor is consistent with mux state: * if mux state is `active`, the neighbor is consistent only if the neighbor is present in `ASIC_DB` but no tunnel route. * if mux state is `standby`, the neighbor is consistent only if the tunnel route is present in `ASIC_DB`1 but no neighbor. 3. if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs. #### How to verify it UT and verify on testbed. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

What I did As the subject. This PR is to leave the vlan neighbor route checking to dualtor_neighbor_check.py script: sonic-net#2840 Signed-off-by: Longxiang Lyu [email protected] How I did it If any misses are found on dualtor, ignore those vlan neighbor misses. How to verify it UT and verify on testbed.

…onic-net#2840) #### What I did Add script `dualtor_neighbor_check.py` to verify the neighbor consistency based on the mux state. It will have the following output: ``` NEIGHBOR MAC PORT MUX_STATE IN_MUX_TOGGLE NEIGHBOR_IN_ASIC TUNNERL_IN_ASIC HWSTATUS ------------- ----------------- ---------- ----------- --------------- ------------------ ----------------- ---------- 192.168.0.2 ee:86:d8:46:7d:01 Ethernet4 standby False no yes consistent 192.168.0.3 86:73:c2:22:bf:02 Ethernet8 standby False no yes consistent 192.168.0.24 56:a6:bf:c5:dd:17 Ethernet92 active False yes no consistent 192.168.0.25 3a:18:56:f5:02:18 Ethernet96 active False yes no consistent 192.168.0.100 00:00:00:00:00:00 N/A N/A N/A no yes consistent ``` Signed-off-by: Longxiang Lyu <[email protected]> #### How I did it the workflow of this scripts: 1. for non-zero-mac neighbors in `APPL_DB` `NEIGH_TABLE`, use the `ASIC_DB` fdb entries to find the mux port that it belongs to. 2. check if the neighbor is consistent with mux state: * if mux state is `active`, the neighbor is consistent only if the neighbor is present in `ASIC_DB` but no tunnel route. * if mux state is `standby`, the neighbor is consistent only if the tunnel route is present in `ASIC_DB`1 but no neighbor. 3. if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs. #### How to verify it UT and verify on testbed. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

lolyu · 2023-11-30T10:57:01Z

Hi @yxieca, could you please help cherry-pick this into 202211 branch?

…2840) #### What I did Add script `dualtor_neighbor_check.py` to verify the neighbor consistency based on the mux state. It will have the following output: ``` NEIGHBOR MAC PORT MUX_STATE IN_MUX_TOGGLE NEIGHBOR_IN_ASIC TUNNERL_IN_ASIC HWSTATUS ------------- ----------------- ---------- ----------- --------------- ------------------ ----------------- ---------- 192.168.0.2 ee:86:d8:46:7d:01 Ethernet4 standby False no yes consistent 192.168.0.3 86:73:c2:22:bf:02 Ethernet8 standby False no yes consistent 192.168.0.24 56:a6:bf:c5:dd:17 Ethernet92 active False yes no consistent 192.168.0.25 3a:18:56:f5:02:18 Ethernet96 active False yes no consistent 192.168.0.100 00:00:00:00:00:00 N/A N/A N/A no yes consistent ``` Signed-off-by: Longxiang Lyu <[email protected]> #### How I did it the workflow of this scripts: 1. for non-zero-mac neighbors in `APPL_DB` `NEIGH_TABLE`, use the `ASIC_DB` fdb entries to find the mux port that it belongs to. 2. check if the neighbor is consistent with mux state: * if mux state is `active`, the neighbor is consistent only if the neighbor is present in `ASIC_DB` but no tunnel route. * if mux state is `standby`, the neighbor is consistent only if the tunnel route is present in `ASIC_DB`1 but no neighbor. 3. if there are any inconsistent neighbors and the mux port is currently in-toggle, the script will have a non-zero negative return, and will write error messages to logs. #### How to verify it UT and verify on testbed. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

bingwang-ms reviewed May 18, 2023

View reviewed changes

scripts/dualtor_neighbor_check.py Outdated Show resolved Hide resolved

bingwang-ms reviewed May 18, 2023

View reviewed changes

scripts/dualtor_neighbor_check.py Outdated Show resolved Hide resolved

bingwang-ms reviewed May 18, 2023

View reviewed changes

scripts/dualtor_neighbor_check.py Outdated Show resolved Hide resolved

prsunny requested a review from Ndancejic May 18, 2023 22:08

Ndancejic previously approved these changes May 19, 2023

View reviewed changes

lolyu dismissed Ndancejic’s stale review via e2dcac1 May 24, 2023 07:23

lolyu force-pushed the add_neighbor_check branch from 2765ec3 to 4639182 Compare May 24, 2023 13:17

lolyu marked this pull request as ready for review May 24, 2023 13:17

lolyu requested review from bingwang-ms, Ndancejic, prsunny, yxieca and lguohan May 24, 2023 13:27

lolyu added Request for 202012 Branch Request for 202205 Branch labels May 29, 2023

lolyu mentioned this pull request Jun 25, 2023

Improve route_check to better handle duplicate updates #2884

Closed

lolyu added 9 commits July 10, 2023 05:47

[dualtor] Add script to verify consistency between kernel and ASIC

c8eec4d

Signed-off-by: Longxiang Lyu <[email protected]>

Add script to copy list

079831d

Signed-off-by: Longxiang Lyu <[email protected]>

Support script load cache and zero mac check

eaea1d3

Signed-off-by: Longxiang Lyu <[email protected]>

Separate check and parse

507de59

Signed-off-by: Longxiang Lyu <[email protected]>

Add UT

5d9e917

Signed-off-by: Longxiang Lyu <[email protected]>

Skip check zero-mac neighbor consistency

ee97c21

Signed-off-by: Longxiang Lyu <[email protected]>

Add UTs to cover log config

eb828e2

Signed-off-by: Longxiang Lyu <[email protected]>

Add UT to cover run_command

ed52539

Signed-off-by: Longxiang Lyu <[email protected]>

Check if port exists

9b3f440

Signed-off-by: Longxiang Lyu <[email protected]>

lolyu force-pushed the add_neighbor_check branch from 709f676 to 9b3f440 Compare July 10, 2023 05:47

Refine error output

21785fe

Signed-off-by: Longxiang Lyu <[email protected]>

Remove shell=True and add redis-cli return check

a6678a7

Signed-off-by: Longxiang Lyu <[email protected]>

lolyu force-pushed the add_neighbor_check branch from 58f1e71 to a6678a7 Compare July 13, 2023 06:59

Merge branch 'master' into add_neighbor_check

ace68bb

Merge branch 'master' into add_neighbor_check

07ecc34

lolyu requested a review from bingwang-ms July 14, 2023 09:25

bingwang-ms reviewed Jul 14, 2023

View reviewed changes

scripts/dualtor_neighbor_check.py Show resolved Hide resolved

bingwang-ms approved these changes Jul 14, 2023

View reviewed changes

lolyu merged commit b617d97 into sonic-net:master Jul 14, 2023

lolyu added Request for 202211 Branch Request for 202305 Branch and removed Request for 202012 Branch labels Jul 14, 2023

yxieca added the Included in 202205 Branch label Jul 21, 2023

This was referenced Jul 25, 2023

[action] [PR:15769] [monit][dualtor] Periodically check mux neighbors consistency sonic-net/sonic-buildimage#15953

Closed

[action] [PR:15769] [monit][dualtor] Periodically check mux neighbors consistency sonic-net/sonic-buildimage#15954

Merged

StormLiangMS added the Included in 202305 Branch label Aug 6, 2023

mssonicbld mentioned this pull request Aug 6, 2023

[action] [PR:15769] [monit][dualtor] Periodically check mux neighbors consistency sonic-net/sonic-buildimage#16055

Merged

11 tasks

yxieca added the Included in 202211 Branch label Nov 30, 2023

lolyu mentioned this pull request Dec 1, 2023

[dualtor_neighbor_check] Adjust zero-mac check condition #3034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dualtor] Add script to verify consistency between kernel and ASIC #2840

[dualtor] Add script to verify consistency between kernel and ASIC #2840

lolyu commented May 17, 2023 •

edited

Loading

yxieca commented May 24, 2023

bingwang-ms commented May 24, 2023

lolyu commented May 25, 2023

StormLiangMS commented Jul 19, 2023

lolyu commented Nov 30, 2023

[dualtor] Add script to verify consistency between kernel and ASIC #2840

[dualtor] Add script to verify consistency between kernel and ASIC #2840

Conversation

lolyu commented May 17, 2023 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

yxieca commented May 24, 2023

bingwang-ms commented May 24, 2023

lolyu commented May 25, 2023

StormLiangMS commented Jul 19, 2023

lolyu commented Nov 30, 2023

lolyu commented May 17, 2023 •

edited

Loading