[201911] : orchagent crash is seen while running vrf tests.

[chitra-raghavan]

Description

in 201911 branch, While running test_vrf.py orchagent crash is seen in Broadcom platform S5232.
Attached syslog .
syslog-vrf.txt
config_db :
config_db_vrf.txt

ed.csv --host-pattern=vms-t0 --module-path=../ansible/library --disable_loganalyzer --skip_sanity -rab -vvv
root@dea27e777114:/var/johnar/sonic-mgmt/tests# vi  test_vrf.py
root@dea27e777114:/var/johnar/sonic-mgmt/tests# py.test test_vrf.py  --testbed=vms-t0 --inventory=/var/johnar/sonic-mgmt/ansible/lab --testbed_fi
le=../ansible/testbed.csv --host-pattern=vms-t0 --module-path=../ansible/library --disable_loganalyzer  --skip_sanity -rab  -vvv
============================================================== test session starts ==============================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
ansible: 2.8.7
rootdir: /var/johnar/sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, xdist-1.28.0, repeat-0.8.0, forked-1.1.3
collected 44 items

test_vrf.py::TestVrfCreateAndBind::test_vrf_in_kernel PASSED                                                                              [  2%]
test_vrf.py::TestVrfCreateAndBind::test_vrf_in_appl_db PASSED                                                                             [  4%]
test_vrf.py::TestVrfCreateAndBind::test_vrf_in_asic_db PASSED                                                                             [  6%]
test_vrf.py::TestVrfNeigh::test_ping_lag_neigh PASSED                                                                                     [  9%]
test_vrf.py::TestVrfNeigh::test_ping_vlan_neigh PASSED                                                                                    [ 11%]
test_vrf.py::TestVrfNeigh::test_vrf1_neigh_ip_fwd PASSED                                                                                  [ 13%]
test_vrf.py::TestVrfNeigh::test_vrf2_neigh_ip_fwd PASSED                                                                                  [ 15%]
test_vrf.py::TestVrfFib::test_show_bgp_summary FAILED                                                                                     [ 18%]
test_vrf.py::TestVrfFib::test_vrf1_fib Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
FAILED                                                                                             [ 20%]
test_vrf.py::TestVrfFib::test_vrf2_fib Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
FAILED                                                                                             [ 22%]
test_vrf.py::TestVrfIsolation::test_neigh_isolate_vrf1_from_vrf2 PASSED                                                                   [ 25%]
test_vrf.py::TestVrfIsolation::test_neigh_isolate_vrf2_from_vrf1 PASSED                                                                   [ 27%]
test_vrf.py::TestVrfIsolation::test_fib_isolate_vrf1_from_vrf2 Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
FAILED                                                                     [ 29%]
test_vrf.py::TestVrfIsolation::test_fib_isolate_vrf2_from_vrf1 Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc
FAILED                                                                     [ 31%]
test_vrf.py::TestVrfAclRedirect::test_origin_ports_recv_no_pkts_v4 Loading callback plugin yaml of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/yaml.pyc


root@sonic-s5232-01:~# gdb /usr/bin/orchagent orchagent.1591211865.39.core
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/orchagent...(no debugging symbols found)...done.
[New LWP 51]
[New LWP 39]
[New LWP 50]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/orchagent -d /var/log/swss -b 8192 -m 3c:2c:30:6d:7e:80'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fbb64b3afff in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fbb640ed700 (LWP 51))]
(gdb) bt
#0  0x00007fbb64b3afff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fbb64b3c42a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fbb654530ad in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007fbb65451066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007fbb654510b1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fbb6547be9e in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fbb666134a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fbb64bf0d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
C(gdb)

Steps to reproduce the issue:

1. Run vrf tests from sonic-mgmt
2. Check for crash.

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

root@sonic-s5232-01:/var/core#
root@sonic-s5232-01:/var/core# show ver

SONiC Software Version: SONiC.HEAD.80-209febcf
Distribution: Debian 9.12
Kernel: 4.9.0-11-2-amd64
Build commit: 209febcf
Build date: Fri May 22 15:33:28 UTC 2020
Built by: johnar@jenkins-worker-3

Platform: x86_64-dellemc_s5232f_c3538-r0
HwSKU: DellEMC-S5232f-C8D48
ASIC: broadcom
Serial Number: CN01WJVTCES0094Q0019
Uptime: 20:15:52 up  1:02,  2 users,  load average: 0.40, 0.38, 0.61

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-syncd-brcm             HEAD.80-209febcf    4b670fe9e96d        432MB
docker-syncd-brcm             latest              4b670fe9e96d        432MB
docker-platform-monitor       HEAD.80-209febcf    2e2672bfd364        337MB
docker-platform-monitor       latest              2e2672bfd364        337MB
docker-sonic-mgmt-framework   HEAD.80-209febcf    ece71aa1e202        425MB
docker-sonic-mgmt-framework   latest              ece71aa1e202        425MB
docker-lldp-sv2               HEAD.80-209febcf    c36883a75a59        307MB
docker-lldp-sv2               latest              c36883a75a59        307MB
docker-dhcp-relay             HEAD.80-209febcf    5f78d0f966cf        295MB
docker-dhcp-relay             latest              5f78d0f966cf        295MB
docker-database               HEAD.80-209febcf    ead5fb0804b9        285MB
docker-database               latest              ead5fb0804b9        285MB
docker-router-advertiser      HEAD.80-209febcf    49a8acb845b3        285MB
docker-router-advertiser      latest              49a8acb845b3        285MB
docker-sflow                  HEAD.80-209febcf    7c8149618e8c        310MB
docker-sflow                  latest              7c8149618e8c        310MB
docker-teamd                  HEAD.80-209febcf    db73c26477c7        310MB
docker-teamd                  latest              db73c26477c7        310MB
docker-snmp-sv2               HEAD.80-209febcf    2304dc30918d        343MB
docker-snmp-sv2               latest              2304dc30918d        343MB
docker-orchagent              HEAD.80-209febcf    f3871b8abfff        328MB
docker-orchagent              latest              f3871b8abfff        328MB

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[rlhui]

@chitra-raghavan - this is not yet supported. There was a PR earlier for this, but it's not merged.
#3047