[VPP-1711] vppctl makes VPP hang-up occasionally

[vvalderrv]

> Description

We found that sometimes VPP itself freeze when we use vppctl command repeatedly, 

This issue doesn't happen immediately but it almost happened in 3-7 days under heavy repeated vppctl command (ex, call vppctl per second)

> VPP version

VPP 18.10 with 2 patches,

• https://gerrit.fd.io/r/#/c/12680/
• https://gerrit.fd.io/r/#/c/18826/

> log & coredump

This is gdb backtrace when VPP hang-up and was stoped by manually HAP.

Please see attached crashdump file in details.

(gdb) bt
#0 0x00007fbdc1f24207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fbdc1f258f8 in __GI_abort () at abort.c:90
#2 0x0000000000407cee in os_exit (code=code@entry=1) at /usr/src/debug/vpp-18.10/src/vpp/vnet/main.c:349
#3 0x00007fbdc35e749c in unix_signal_handler (signum=<optimized out>, si=<optimized out>, uc=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/unix/main.c:157
#4 <signal handler called>
#5 0x00007fbdc1fd0d47 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#6 0x00007fbdc2d0a0be in spin_acquire_lock (sl=sl@entry=0x7fbd808a5384) at /usr/src/debug/vpp-18.10/src/vppinfra/dlmalloc.c:466
#7 0x00007fbdc2d0b960 in mspace_malloc (msp=0x7fbd808a5010, bytes=bytes@entry=13) at /usr/src/debug/vpp-18.10/src/vppinfra/dlmalloc.c:4339
#8 0x00007fbdc2d0d1c8 in mspace_get_aligned (msp=0x7fbd808a5010, n_user_data_bytes=13, n_user_data_bytes@entry=9, align=align@entry=8, align_offset=align_offset@entry=8)
at /usr/src/debug/vpp-18.10/src/vppinfra/dlmalloc.c:4178
#9 0x00007fbdc2d02880 in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=8, align=8, size=9) at /usr/src/debug/vpp-18.10/src/vppinfra/mem.h:118
#10 vec_resize_allocate_memory (v=v@entry=0x0, length_increment=length_increment@entry=1, data_bytes=9, header_bytes=<optimized out>, header_bytes@entry=0,
data_align=data_align@entry=8) at /usr/src/debug/vpp-18.10/src/vppinfra/vec.c:59
#11 0x00007fbdc35ddf73 in _vec_resize_inline (data_align=<optimized out>, header_bytes=<optimized out>, data_bytes=<optimized out>, length_increment=<optimized out>, v=<optimized out>)
at /usr/src/debug/vpp-18.10/src/vppinfra/vec.h:145
#12 unix_cli_add_pending_output (uf=0x7fbd8181238c, buffer=0x7fbdc35efa6d "\r", buffer_bytes=1, cf=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/unix/cli.c:544
#13 0x00007fbdc35e0e50 in unix_cli_file_welcome (cf=0x7fbd819af9cc, cm=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/unix/cli.c:1216
#14 0x00007fbdc2cebb6c in timer_interrupt (signum=<optimized out>) at /usr/src/debug/vpp-18.10/src/vppinfra/timer.c:129
#15 <signal handler called>
#16 0x00007fbdc2d0b866 in mspace_malloc (msp=0x7fbd808a5010, bytes=bytes@entry=20) at /usr/src/debug/vpp-18.10/src/vppinfra/dlmalloc.c:4355
#17 0x00007fbdc2d0d1c8 in mspace_get_aligned (msp=0x7fbd808a5010, n_user_data_bytes=20, n_user_data_bytes@entry=16, align=align@entry=8, align_offset=align_offset@entry=8)
at /usr/src/debug/vpp-18.10/src/vppinfra/dlmalloc.c:4178
#18 0x00007fbdc2d02880 in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=8, align=8, size=16) at /usr/src/debug/vpp-18.10/src/vppinfra/mem.h:118
#19 vec_resize_allocate_memory (v=v@entry=0x0, length_increment=1, data_bytes=16, data_bytes@entry=8, header_bytes=<optimized out>, header_bytes@entry=0, data_align=data_align@entry=8)
at /usr/src/debug/vpp-18.10/src/vppinfra/vec.c:59
#20 0x00007fbdc35e1f70 in _vec_resize_inline (data_align=8, header_bytes=0, data_bytes=8, length_increment=1, v=<optimized out>) at /usr/src/debug/vpp-18.10/src/vppinfra/vec.h:145
#21 vlib_process_get_events (data_vector=<synthetic pointer>, vm=0x7fbdc37fd200 <vlib_global_main>) at /usr/src/debug/vpp-18.10/src/vlib/node_funcs.h:562
#22 unix_cli_process (vm=0x7fbdc37fd200 <vlib_global_main>, rt=0x7fbd819d6000, f=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/unix/cli.c:2530
#23 0x00007fbdc35ba126 in vlib_process_bootstrap (_a=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/main.c:1232
#24 0x00007fbdc2cc26a4 in clib_calljmp () from /lib64/libvppinfra.so.18.10
#25 0x00007fbd80fffbd0 in ?? ()
#26 0x00007fbdc35bb299 in dispatch_process (vm=0x7fbdc37fd200 <vlib_global_main>, p=0x7fbd819d6000, last_time_stamp=0, f=0x0) at /usr/src/debug/vpp-18.10/src/vlib/main.c:1254
#27 0x0000000000000000 in ?? ()
(gdb) 

Thanks

Tatsumi.