orchagent crashed when adding the members, addresses from an existing portchannel to a newly created portchannel

[mannytaheri]

Description

orchagent crashed in testcase test_po_update_io_no_loss[lc]::teardown. This issue is also reproducible manually
The orchagent crash is caused by:

1. Removing the members and ipv4/ipv6 address of an existing PortChannel
2. Then creating a new Portchannel and adding the members, ipv4/ipv6 address from the
   old Portchannel to the newly created portchannel .

Please see syslog attached.

syslog.txt

Jan 3 21:40:15.387019 ixre-egl-board3 NOTICE swss1#orchagent: :- removeRouterIntfs: Remove router interface for port PortChannel106
Jan 3 21:40:15.492690 ixre-egl-board3 NOTICE swss1#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.1@Ethernet-IB1,10.0.0.5@Ethernet-IB1,10.0.0.7@PortChannel999,10.0.0.11@Ethernet184
Jan 3 21:40:15.521978 ixre-egl-board3 NOTICE swss1#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.7@PortChannel999,10.0.0.11@Ethernet184
Jan 3 21:40:15.711079 ixre-egl-board3 NOTICE swss0#orchagent: :- removeNextHopGroup: Delete next hop group fc00::2@PortChannel102,fc00::a@Ethernet64,fc00::e@Ethernet-IB0,fc00::16@Ethernet-IB0
Jan 3 21:40:15.717072 ixre-egl-board3 NOTICE swss0#orchagent: :- removeNextHopGroup: Delete next hop group fc00::e@Ethernet-IB0,fc00::16@Ethernet-IB0
Jan 3 21:40:15.749977 ixre-egl-board3 NOTICE swss0#orchagent: :- addLag: Create an empty LAG ixre-egl-board3|asic1|PortChannel999 lid:2000000000a95
Jan 3 21:40:15.754730 ixre-egl-board3 NOTICE syncd0#syncd: :- removeRif: Trying to remove nonexisting router interface counter from Id 0x6000000000847
Jan 3 21:40:15.755829 ixre-egl-board3 ERR swss0#orchagent: :- meta_generic_validation_remove: object 0x6000000000847 reference count is 4, can't remove
Jan 3 21:40:15.756968 ixre-egl-board3 ERR swss0#orchagent: :- removeRouterIntfs: Failed to remove router interface for port ixre-egl-board3|asic1|PortChannel106, rv:-17
Jan 3 21:40:15.758146 ixre-egl-board3 ERR swss0#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_ROUTER_INTERFACE, status: SAI_STATUS_OBJECT_IN_USE

Steps to reproduce the issue:

1. sudo config portchannel -n asic1 member del PortChannel106 Ethernet208
2. sudo config portchannel -n asic1 member del PortChannel106 Ethernet216
3. sudo config interface -n asic1 ip remove PortChannel106 10.0.0.6/31
4. sudo config interface -n asic1 ip remove PortChannel106 fc00::d/126
5. sudo config portchannel -n asic1 add PortChannel999
6. sudo config portchannel -n asic1 member add PortChannel999 Ethernet208
7. sudo config portchannel -n asic1 member add PortChannel999 Ethernet216
8. sudo config interface -n asic1 ip add PortChannel999 10.0.0.6/31
9. sudo config interface -n asic1 ip add PortChannel999 fc00::d/126

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[prsunny]

Do you've any delays between the above commands or copy-paste the whole set? Would you try with a sleep of 3-4 sec between step 4 and step 5 and share the result?

[linqingxuan]

In which version did you encounter this problem? Please add a screenshot or text of "show version"

[mannytaheri]

show ver

SONiC Software Version: SONiC.20220532.54
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: b9e6caad98
Build date: Tue Jan 9 00:13:06 UTC 2024
Built by: cloudtest@95bebd0dc000000

[volodymyrsamotiy]

Fix will be in sonic-mgmt test as well as in swss to add some protection.
@prsunny, will discuss with swss owners and further triage.

[saksarav-nokia]

@prsunny , we had the orchagent crash with same signature when we remove the ports from Portchannel and remove the Ip and ipv6 address from Portchannel. What i noticed in the log/code is that the addNeighbor adds the remote system neighbor against the remote system port and increment the RIF reference counter for remote system port. However when it adds the nextHop in addNextHop , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port. When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. I think the SWSS PR sonic-net/sonic-swss#1686 made this change in addNextHop as part of Mpls support. Since RIF-If od remote system port is used for nexthop, we should be increasing the ref count for remote system port in addNextHop right?. Please let me know your thoughts on this.

[prsunny]

I see. Would you provide a fix PR?

[saksarav-nokia]

Yes. We are testing the fix for both Port channel scenarios and once we confirm that the fix is working, i will create a PR.

[saksarav-nokia]

Created PR sonic-net/sonic-swss#3042

[kenneth-arista]

This is the same as #17204