Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tlm_teamd crash when config a new portchannel #5306

Closed
tim-rj opened this issue Sep 3, 2020 · 2 comments · Fixed by sonic-net/sonic-swss#1476
Closed

tlm_teamd crash when config a new portchannel #5306

tim-rj opened this issue Sep 3, 2020 · 2 comments · Fixed by sonic-net/sonic-swss#1476
Assignees
@pavel-shirshov

Comments

@tim-rj
Copy link
Contributor

tim-rj commented Sep 3, 2020

Description

tlm_teamd crash when config a new portchannel

Steps to reproduce the issue:
1.config a new portchannel as follow



root@sonic:/home/admin# show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  -------------
 0001  PortChannel0001  LACP(A)(Up)  Ethernet51(S)
 0002  PortChannel0002  LACP(A)(Dw)  N/A
root@sonic:/home/admin# config portchannel add PortChannel0003

Describe the results you received:
tlm_teamd crash and generate the core :
syslog as follow

root@sonic:/home/admin# show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  -------------
 0001  PortChannel0001  LACP(A)(Up)  Ethernet51(S)
 0002  PortChannel0002  LACP(A)(Dw)  N/A
root@sonic:/home/admin# config portchannel add PortChannel0003
root@sonic:/home/admin# show interfaces portchannel           
Traceback (most recent call last):
  File "/usr/bin/show", line 12, in <module>
    sys.exit(cli())
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/show/interfaces/portchannel.py", line 156, in portchannel
    team.get_teamshow_result()
  File "/usr/lib/python2.7/dist-packages/show/interfaces/portchannel.py", line 100, in get_teamshow_result
    info['protocol'] += "(A)" if state['runner.active'] == "true" else '(I)'
KeyError: 'runner.active'
root@sonic:/home/admin# show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol    Ports
-----  ---------------  ----------  -------
 0001  PortChannel0001  N/A
 0002  PortChannel0002  N/A
 0003  PortChannel0003  N/A
root@sonic:/home/admin# sudo tail -n 20 /var/log/syslog
Sep  3 10:52:45.352710 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:17 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:0 type:team
Sep  3 10:52:45.387919 sonic INFO teamd#supervisord: teammgrd Device "PortChannel0001" does not exist
Sep  3 10:52:46.105959 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.105959 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.110093 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.110093 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.281284 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.281284 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.331826 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.331826 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.332067 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.332067 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:47.110134 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:47.110134 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:47.281292 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:47.281292 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:48.110096 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:48.110096 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:48.281171 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:48.281171 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
root@sonic:/home/admin# sudo tail -n 200 /var/log/syslog
Sep  3 10:49:09.978800 sonic ERR route_check.py:  results: { {#012    "Unaccounted_ROUTE_ENTRY_TABLE_entries": [#012        "0.0.0.0/0", #012        "::/0"#012    ]#012} }
Sep  3 10:49:09.979253 sonic ERR route_check.py:  Failed. Look at reported mismatches above
Sep  3 10:50:09.577244 sonic ERR monit[546]: 'routeCheck' status failed (255) -- results: { {#012    "Unaccounted_ROUTE_ENTRY_TABLE_entries": [#012        "0.0.0.0/0", #012        "::/0"#012    ]#e
Sep  3 10:50:09.583816 sonic ERR monit[546]: 'telemetry' process is not running
Sep  3 10:50:09.590806 sonic ERR monit[546]: 'dialout_client' process is not running
Sep  3 10:50:09.597810 sonic ERR monit[546]: 'sflowmgrd' process is not running
Sep  3 10:50:28.580512 sonic INFO dhclient[1809]: XMT: Solicit on eth0, interval 120290ms.
Sep  3 10:51:09.613868 sonic ERR monit[546]: 'telemetry' process is not running
Sep  3 10:51:09.619256 sonic ERR monit[546]: 'dialout_client' process is not running
Sep  3 10:51:09.625882 sonic ERR monit[546]: 'sflowmgrd' process is not running
Sep  3 10:52:09.651297 sonic ERR monit[546]: 'telemetry' process is not running
Sep  3 10:52:09.656878 sonic ERR monit[546]: 'dialout_client' process is not running
Sep  3 10:52:09.664117 sonic ERR monit[546]: 'sflowmgrd' process is not running
Sep  3 10:52:28.968389 sonic INFO dhclient[1809]: XMT: Solicit on eth0, interval 113600ms.
Sep  3 10:52:31.166139 sonic INFO teamd#supervisord: teammgrd Using team device "PortChannel0003".
Sep  3 10:52:31.166139 sonic INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel0003.pid"
Sep  3 10:52:31.166139 sonic INFO teamd#supervisord: teammgrd This program is not intended to be run as root.
Sep  3 10:52:31.168911 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:0 oper:0 addr:f2:aa:98:dc:c7:19 ifindex:70 master:0 type:team
Sep  3 10:52:31.172214 sonic INFO systemd-udevd[5276]: Using default interface naming scheme 'v240'.
Sep  3 10:52:31.172448 sonic INFO systemd-udevd[5276]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep  3 10:52:31.172601 sonic WARNING systemd-udevd[5276]: Could not generate persistent MAC address for PortChannel0003: No such file or directory
Sep  3 10:52:31.172831 sonic NOTICE swss#orchagent: message repeated 1594 times: [ :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS]
Sep  3 10:52:31.172831 sonic NOTICE swss#orchagent: :- addLag: Create an empty LAG PortChannel0003 lid:2000000000a12
Sep  3 10:52:31.172831 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:0 oper:0 addr:f2:aa:98:dc:c7:19 ifindex:70 master:0 type:team
Sep  3 10:52:31.172831 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:70 master:0 type:team
Sep  3 10:52:31.174160 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:31.175205 sonic INFO kernel: [ 1106.803219] PortChannel0003: Mode changed to "loadbalance"
Sep  3 10:52:31.179921 sonic NOTICE teamd#teammgrd: :- addLag: Start port channel PortChannel0003 with teamd
Sep  3 10:52:31.182369 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:1 oper:0 addr:58:69:6c:fb:21:22 ifindex:70 master:0 type:team
Sep  3 10:52:31.183388 sonic INFO kernel: [ 1106.812518] IPv6: ADDRCONF(NETDEV_UP): PortChannel0003: link is not ready
Sep  3 10:52:31.183402 sonic INFO kernel: [ 1106.812526] 8021q: adding VLAN 0 to HW filter on device PortChannel0003
Sep  3 10:52:31.183971 sonic NOTICE teamd#teammgrd: :- setLagAdminStatus: Set port channel PortChannel0003 admin status to up
Sep  3 10:52:31.188278 sonic NOTICE teamd#teammgrd: :- setLagMtu: Set port channel PortChannel0003 MTU to 9100
Sep  3 10:52:32.175287 sonic INFO kernel: [ 1107.803909] tlm_teamd[2937]: segfault at 0 ip 00005651fc9b2b58 sp 00007fffe99ec650 error 4 in tlm_teamd[5651fc9af000+9000]
Sep  3 10:52:32.175335 sonic INFO kernel: [ 1107.803923] Code: 8b 74 24 20 4c 89 e7 48 8b 54 24 28 89 44 24 40 49 8d 44 24 10 48 01 f2 48 89 04 24 e8 e1 09 00 00 48 8b 34 24 49 8d 44 24 10 <48> 8b6
Sep  3 10:52:33.266065 sonic INFO teamd#supervisor-proc-exit-listener: Process tlm_teamd exited unxepectedly. Terminating supervisor...
Sep  3 10:52:34.267550 sonic NOTICE teamd#teamsyncd: :- cleanTeamSync: Cleaning up LAG teamd resources ...
Sep  3 10:52:34.268555 sonic NOTICE swss#orchagent: message repeated 11 times: [ :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS]
Sep  3 10:52:34.268555 sonic NOTICE swss#orchagent: :- removeLagMember: Remove member Ethernet51 from LAG PortChannel0001 lid:20000000009b6 lmid:1b0000000009d2
Sep  3 10:52:34.268555 sonic NOTICE swss#orchagent: :- setHostIntfsStripTag: Set SAI_HOSTIF_VLAN_TAG_STRIP to host interface: Ethernet51
Sep  3 10:52:34.268984 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:34.269930 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.270068 sonic NOTICE swss#orchagent: :- removeLag: Remove LAG PortChannel0002 lid:20000000009b7
Sep  3 10:52:34.270479 sonic NOTICE swss#orchagent: :- removeLag: Remove LAG PortChannel0003 lid:2000000000a12
Sep  3 10:52:34.270575 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.270872 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:34.271054 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.271103 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.271103 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:34.271359 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.271359 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.271359 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:34.281254 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:34.281254 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:35.110203 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:35.110203 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:35.281305 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:35.281305 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:36.105837 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:36.105837 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:36.109886 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:36.109886 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:36.281154 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:36.281154 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:36.633602 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:36.633602 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:37.110160 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:37.110160 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:37.281251 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:37.281251 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:38.110146 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:38.110146 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:38.281277 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:38.281277 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:39.082858 sonic INFO teamd#supervisord 2020-09-03 10:52:32,256 INFO exited: tlm_teamd (terminated by SIGSEGV (core dumped); not expected)
Sep  3 10:52:39.082858 sonic INFO teamd#supervisord 2020-09-03 10:52:33,265 WARN received SIGTERM indicating exit request
Sep  3 10:52:39.082858 sonic INFO teamd#supervisord 2020-09-03 10:52:33,266 INFO waiting for teammgrd, teamsyncd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:39.082858 sonic INFO teamd#supervisord 2020-09-03 10:52:36,270 INFO waiting for teammgrd, teamsyncd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:39.110082 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:39.110082 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:39.281099 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:39.281099 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:40.110095 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:40.110095 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:40.281277 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:40.281277 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:41.110121 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:41.110121 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:41.281240 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:41.281240 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:42.110080 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:42.110080 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:42.281108 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:42.281108 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:43.110130 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:43.110130 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:43.281289 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:43.281289 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:44.110127 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:44.110127 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:44.281309 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:44.281309 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.110100 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:45.110100 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.281256 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:45.281256 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.285071 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Cleaning up LAGs during shutdown...
Sep  3 10:52:45.285579 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:1 oper:0 addr:58:69:6c:fb:21:22 ifindex:70 master:0 type:team
Sep  3 10:52:45.285744 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:1 oper:1 addr:58:69:6c:fb:21:22 ifindex:61 master:8
Sep  3 10:52:45.286145 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:1 oper:1 addr:58:69:6c:fb:21:22 ifindex:61 master:8
Sep  3 10:52:45.286145 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:1 oper:1 addr:58:69:6c:fb:21:22 ifindex:61 master:0
Sep  3 10:52:45.286643 sonic NOTICE swss#portsyncd: :- onMsg: Publish Ethernet51(ok) to state db
Sep  3 10:52:45.286806 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:61 master:0
Sep  3 10:52:45.287835 sonic NOTICE swss#portsyncd: :- onMsg: Publish Ethernet51(ok) to state db
Sep  3 10:52:45.287989 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:61 master:0
Sep  3 10:52:45.288085 sonic INFO kernel: [ 1120.915578] device Ethernet51 left promiscuous mode
Sep  3 10:52:45.288111 sonic INFO kernel: [ 1120.917136] PortChannel0001: Port device Ethernet51 removed
Sep  3 10:52:45.288536 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:45.288536 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.288566 sonic NOTICE swss#portsyncd: :- onMsg: Publish Ethernet51(ok) to state db
Sep  3 10:52:45.288768 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:61 master:0
Sep  3 10:52:45.288991 sonic NOTICE swss#portsyncd: :- onMsg: Publish Ethernet51(ok) to state db
Sep  3 10:52:45.289068 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet51 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:61 master:0
Sep  3 10:52:45.289442 sonic NOTICE swss#portsyncd: :- onMsg: Publish Ethernet51(ok) to state db
Sep  3 10:52:45.289501 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:1 oper:1 addr:58:69:6c:fb:21:22 ifindex:8 master:66 type:team
Sep  3 10:52:45.289584 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0002 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:9 master:0 type:team
Sep  3 10:52:45.289651 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:17 key:PortChannel0002 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:9 master:0 type:team
Sep  3 10:52:45.291659 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0003 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:70 master:0 type:team
Sep  3 10:52:45.293507 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:17 key:PortChannel0003 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:70 master:0 type:team
Sep  3 10:52:45.296664 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:1 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66
Sep  3 10:52:45.296687 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:1 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66 type:team
Sep  3 10:52:45.296785 sonic INFO kernel: [ 1120.925004] Bridge: port 3(PortChannel0001) entered disabled state
Sep  3 10:52:45.298181 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:45.298181 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.298242 sonic NOTICE syncd#syncd: :- processBulkQuadEvent: bulk SAI_OBJECT_TYPE_ROUTE_ENTRY execute with 1 items
Sep  3 10:52:45.298272 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:45.298272 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:45.316113 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66 type:team
Sep  3 10:52:45.316309 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66
Sep  3 10:52:45.316471 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66
Sep  3 10:52:45.316471 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:17 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:66
Sep  3 10:52:45.319223 sonic INFO kernel: [ 1120.946175] Bridge: port 3(PortChannel0001) entered disabled state
Sep  3 10:52:45.319252 sonic INFO kernel: [ 1120.946467] device PortChannel0001 left promiscuous mode
Sep  3 10:52:45.319256 sonic INFO kernel: [ 1120.946470] Bridge: port 3(PortChannel0001) entered disabled state
Sep  3 10:52:45.319962 sonic INFO teamd#supervisord: teammgrd libteamdctl: teamdctl_connect: Failed to connect using all CLIs.
Sep  3 10:52:45.319962 sonic INFO teamd#supervisord: teammgrd teamdctl_connect failed (Invalid argument)
Sep  3 10:52:45.320397 sonic ERR teamd#teammgrd: :- addLagMember: Failed to add Ethernet51 to port channel PortChannel0001
Sep  3 10:52:45.352710 sonic NOTICE swss#portsyncd: :- onMsg: nlmsg type:17 key:PortChannel0001 admin:0 oper:0 addr:58:69:6c:fb:21:22 ifindex:8 master:0 type:team
Sep  3 10:52:45.387919 sonic INFO teamd#supervisord: teammgrd Device "PortChannel0001" does not exist
Sep  3 10:52:46.105959 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.105959 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.110093 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.110093 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.281284 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.281284 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.331826 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.331826 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:46.332067 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:46.332067 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:47.110134 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:47.110134 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:47.281292 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:47.281292 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:48.110096 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:48.110096 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:48.281171 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:48.281171 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:39,275 INFO waiting for teammgrd, teamsyncd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:42,279 INFO waiting for teammgrd, teamsyncd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:43,281 WARN killing 'teamsyncd' (40) with SIGKILL
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:43,282 INFO stopped: teamsyncd (terminated by SIGKILL)
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:45,285 INFO waiting for teammgrd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:45,354 INFO reaped unknown pid 35
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:45,354 INFO reaped unknown pid 50
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:45,392 INFO reaped unknown pid 27
Sep  3 10:52:49.092855 sonic INFO teamd#supervisord 2020-09-03 10:52:48,397 INFO waiting for teammgrd, supervisor-proc-exit-listener, rsyslogd to die
Sep  3 10:52:49.109984 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:49.109984 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:49.281174 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:49.281174 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:50.110101 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:50.110101 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:50.281161 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:50.281161 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:51.110099 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:51.110099 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:51.281192 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:51.281192 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:52.110097 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:52.110097 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:52.281199 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:52.281199 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:53.110074 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:53.110074 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:53.281258 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:53.281258 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:54.110103 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:54.110103 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS
Sep  3 10:52:54.281222 sonic ERR swss#orchagent: :- removeLag: Failed to remove LAG PortChannel0001, it is still in VLAN
Sep  3 10:52:54.281222 sonic NOTICE swss#orchagent: :- set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS

Describe the results you expected:
config success and no print ERR syslog

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

```

Sep 1 17:16:13.229631 sonic ERR monit[555]: 'sflowmgrd' process is not running
root@sonic:/home/admin# show version

SONiC Software Version: SONiC.master.399-07b9d7f4
Distribution: Debian 10.5
Kernel: 4.19.0-9-2-amd64
Build commit: 07b9d7f
Build date: Sun Aug 30 07:41:30 UTC 2020
Built by: johnar@jenkins-worker-8

Platform: x86_64-ruijie_b6510-48vs8cq-r0
HwSKU: B6510-48VS8CQ
ASIC: broadcom
Serial Number: G1W10072
Uptime: 17:26:38 up 2:51, 1 user, load average: 0.46, 0.37, 0.29

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-teamd latest 0da13af040db 386MB
docker-teamd master.399-07b9d7f4 0da13af040db 386MB
docker-sonic-mgmt-framework latest b3309c50c848 481MB
docker-sonic-mgmt-framework master.399-07b9d7f4 b3309c50c848 481MB
docker-router-advertiser latest 81563660196e 355MB
docker-router-advertiser master.399-07b9d7f4 81563660196e 355MB
docker-platform-monitor latest bfc946290aa4 429MB
docker-platform-monitor master.399-07b9d7f4 bfc946290aa4 429MB
docker-lldp latest 17607fd34cc8 383MB
docker-lldp master.399-07b9d7f4 17607fd34cc8 383MB
docker-dhcp-relay latest 57d5a7e2d58d 362MB
docker-dhcp-relay master.399-07b9d7f4 57d5a7e2d58d 362MB
docker-database latest ddb145fe6d62 355MB
docker-database master.399-07b9d7f4 ddb145fe6d62 355MB
docker-orchagent latest 2b0a4c92794f 400MB
docker-orchagent master.399-07b9d7f4 2b0a4c92794f 400MB
docker-nat latest eb5c6b6736c2 389MB
docker-nat master.399-07b9d7f4 eb5c6b6736c2 389MB
docker-sonic-telemetry latest 6c4292e948fd 425MB
docker-sonic-telemetry master.399-07b9d7f4 6c4292e948fd 425MB
docker-fpm-frr latest 076744e3f438 402MB
docker-fpm-frr master.399-07b9d7f4 076744e3f438 402MB
docker-sflow latest 1046ede2b365 390MB
docker-sflow master.399-07b9d7f4 1046ede2b365 390MB
docker-snmp latest ffbf20d1540a 395MB
docker-snmp master.399-07b9d7f4 ffbf20d1540a 395MB
docker-syncd-brcm latest d62f6c218ce2 447MB
docker-syncd-brcm master.399-07b9d7f4 d62f6c218ce2 447MB

root@sonic:/home/admin#
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
@tim-rj
Copy link
Contributor Author

tim-rj commented Sep 3, 2020

root@sonic:/var/core# ls
tlm_teamd.1599130352.21.core.gz
root@sonic:/var/core# ls -al
total 208
drwxr-xr-x 1 root root 4096 Sep 3 10:52 .
drwxr-xr-x 1 root root 4096 Sep 1 17:30 ..
-rw-rw-rw- 1 root root 202339 Sep 3 10:52 tlm_teamd.1599130352.21.core.gz
root@sonic:/var/core#

@OlaOliynyk OlaOliynyk mentioned this issue Sep 7, 2020
Closed
@akokhan akokhan mentioned this issue Sep 29, 2020
Closed
@pavel-shirshov pavel-shirshov self-assigned this Oct 22, 2020
@pavel-shirshov
Copy link
Contributor

Something wrong with zmq initialization

root@str-s6100-acs-1:/# gdb -core /tlm_teamd.1603426041.21.core /usr/bin/tlm_teamd    
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/tlm_teamd...Reading symbols from /usr/lib/debug/.build-id/ad/07742ca6197405a185865f803a8ba312431bfa.debug...done.
done.
[New LWP 21]
[New LWP 23]
[New LWP 103]
[New LWP 102]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/tlm_teamd'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f06d5a8c7bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f06d529dc40 (LWP 21))]
(gdb) bt
#0  0x00007f06d5a8c7bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f06d5a77535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f06d588d773 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#3  0x00007f06d58d42b8 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#4  0x00007f06d58d53e5 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#5  0x00007f06d5f67647 in cli_zmq_init (tdc=0x5632efb3c460, team_name=<optimized out>, 
    priv=0x5632efb436b0) at cli_zmq.c:221
#6  0x00007f06d5f65b07 in cli_init (team_name=0x5632efb2df78 "PortChannel0012", tdc=0x5632efb3c460)
    at libteamdctl.c:262
#7  teamdctl_connect (tdc=0x5632efb3c460, team_name=0x5632efb2df78 "PortChannel0012", 
    addr=addr@entry=0x0, cli_type=cli_type@entry=0x0) at libteamdctl.c:336
#8  0x00005632ef255936 in TeamdCtlMgr::try_add_lag (this=this@entry=0x7ffff83afad0, 
    lag_name="PortChannel0012") at /usr/include/c++/8/bits/basic_string.h:2290
#9  0x00005632ef255a82 in TeamdCtlMgr::add_lag (this=this@entry=0x7ffff83afad0, 
    lag_name="PortChannel0012") at teamdctl_mgr.cpp:66
#10 0x00005632ef25350d in update_interfaces (table=..., mgr=...) at main.cpp:35
#11 0x00005632ef25307b in main () at main.cpp:100

@pavel-shirshov pavel-shirshov mentioned this issue Oct 23, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pavel-shirshov pavel-shirshov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[tlm_teamd]: Force tlm_teamd to connect using usock method only pavel-shirshov/sonic-swss
2 participants
@pavel-shirshov @tim-rj