zebra: Free up leaked memory in zebra_ptm.c

[donaldsharp]

ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16416 byte(s) in 1 object(s) allocated from:
#0 0x7f08e3ca8602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
#1 0x7f08e389c4b0 in qmalloc lib/memory.c:105
#2 0x7f08e38e87b4 in stream_new lib/stream.c:106
#3 0x481d7f in _zebra_ptm_bfd_client_deregister zebra/zebra_ptm.c:1348
#4 0x4e7b84 in hook_call_zserv_client_close zebra/zserv.c:544
#5 0x4e7b84 in zserv_client_free zebra/zserv.c:560
#6 0x4e7b84 in zserv_close_client zebra/zserv.c:625
#7 0x4e7fe0 in zserv_handle_client_fail zebra/zserv.c:638
#8 0x7f08e3901995 in thread_call lib/thread.c:1549
#9 0x7f08e38937d7 in frr_run lib/libfrr.c:1093
#10 0x41686e in main zebra/main.c:470
#11 0x7f08e2fe682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Direct leak of 16416 byte(s) in 1 object(s) allocated from:
#0 0x7f08e3ca8602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
#1 0x7f08e389c4b0 in qmalloc lib/memory.c:105
#2 0x7f08e38e87b4 in stream_new lib/stream.c:106
#3 0x481efe in _zebra_ptm_reroute zebra/zebra_ptm.c:1411
#4 0x4f7dc9 in zserv_handle_commands zebra/zapi_msg.c:2642
#5 0x4e6d32 in zserv_process_messages zebra/zserv.c:517
#6 0x7f08e3901995 in thread_call lib/thread.c:1549
#7 0x7f08e38937d7 in frr_run lib/libfrr.c:1093
#8 0x41686e in main zebra/main.c:470
#9 0x7f08e2fe682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: 32832 byte(s) leaked in 2 allocation(s).

This commit fixes these two different leaks.

Fixes: #5658
Signed-off-by: Donald Sharp [email protected]

[LabN-CI]

💚 Basic BGPD CI results: SUCCESS, 0 tests failed

Results table

_	_
Result	SUCCESS git merge/5659 d13785b
Date	01/08/2020
Start	19:45:21
Finish	20:11:10
Run-Time	25:49
Total	1815
Pass	1815
Fail	0
Valgrind-Errors	0
Valgrind-Loss	0
Details	vncregress-2020-01-08-19:45:21.txt
Log	autoscript-2020-01-08-19:46:18.log.bz2
Memory	421 412 360

For details, please contact louberger

[NetDEF-CI]

Continuous Integration Result: FAILED

See below for issues.
CI System Testrun URL: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

Get source / Pull Request: Successful

Building Stage: Successful

Basic Tests: Failed

Topology tests on Ubuntu 18.04 amd64: Failed (click for details)

Topology Test Results are at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOU1804-10357/test

Topology Tests failed for Topology tests on Ubuntu 18.04 amd64:

2020-01-09 01:14:34,214 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:34,858 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:35,499 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:36,119 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:36,738 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:37,360 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:37,991 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:38,610 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:39,252 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:39,885 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:40,519 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:41,153 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:41,777 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:42,411 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:43,036 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:43,667 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:44,315 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:44,947 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:45,571 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:46,199 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:46,820 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:47,452 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:48,080 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:48,702 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:49,330 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:49,978 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:50,636 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:51,257 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:51,889 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:52,515 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:53,146 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:53,775 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:54,426 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:55,050 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:55,707 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:56,334 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:56,978 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:57,606 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:58,229 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:58,851 WARNING: vtysh_cmd: failed to convert json output
2020-01-09 01:14:59,355 ERROR: 'router_json_cmp' failed after 25.29 seconds
2020-01-09 01:14:59,359 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_protocols_convergence": "r1" JSON output mismatches
assert expected key(s) [u'2001:db8:4::/64', u'2001:db8:6::/64', u'2001:db8:7::/64', u'2001:db8:1::/64'] in json (have []):
  --- Expected value
  +++ Current value
  @@ -1,88 +1 @@
  -{
  -    "2001:db8:1::/64": [
  -        {
  -            "destSelected": true, 
  -            "distance": 0, 
  -            "metric": 0, 
  -            "nexthops": [
  -                {
  -                    "active": true, 
  -                    "directlyConnected": true, 
  -                    "fib": true, 
  -                    "flags": 3, 
  -                    "interfaceName": "r1-eth0"
  -                }
  -            ], 
  -            "prefix": "2001:db8:1::/64", 
  -            "protocol": "connected", 
  -            "selected": true
  -        }
  -    ], 
  -    "2001:db8:4::/64": [
  -        {
  -            "destSelected": true, 
  -            "distance": 1, 
  -            "metric": 0, 
  -            "nexthops": [
  -                {
  -                    "active": true, 
  -                    "afi": "ipv6", 
  -                    "fib": true, 
  -                    "flags": 3, 
  -                    "interfaceName": "r1-eth0"
  -                }
  -            ], 
  -            "prefix": "2001:db8:4::/64", 
  -            "protocol": "static", 
  -            "selected": true
  -        }
  -    ], 
  -    "2001:db8:6::/64": [
  -        {
  -            "destSelected": true, 
  -            "distance": 20, 
  -            "metric": 0, 
  -            "nexthops": [
  -                {
  -                    "active": true, 
  -                    "afi": "ipv6", 
  -                    "ip": "2001:db8:4::1", 
  -                    "recursive": true
  -                }, 
  -                {
  -                    "afi": "ipv6", 
  -                    "fib": true, 
  -                    "interfaceName": "r1-eth0", 
  -                    "ip": "2001:db8:1::2"
  -                }
  -            ], 
  -            "prefix": "2001:db8:6::/64", 
  -            "protocol": "bgp", 
  -            "selected": true
  -        }
  -    ], 
  -    "2001:db8:7::/64": [
  -        {
  -            "destSelected": true, 
  -            "distance": 20, 
  -            "metric": 0, 
  -            "nexthops": [
  -                {
  -                    "active": true, 
  -                    "afi": "ipv6", 
  -                    "ip": "2001:db8:4::1", 
  -                    "recursive": true
  -                }, 
  -                {
  -                    "afi": "ipv6", 
  -                    "fib": true, 
  -                    "interfaceName": "r1-eth0", 
  -                    "ip": "2001:db8:1::2"
  -                }
  -            ], 
  -            "prefix": "2001:db8:7::/64", 
  -            "protocol": "bgp", 
  -            "selected": true
  -        }
  -    ]
  -}
  +{}
2020-01-09 01:15:04,639 ERROR: 'router_json_cmp' failed after 5.10 seconds
2020-01-09 01:15:04,644 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_bfd_comes_back_again": "r1" JSON output mismatches
assert json too few items (have 0, expected 1:
   --- Expected value
  +++ Current value
  @@ -1,16 +1 @@
  -[
  -    {
  -        "diagnostic": "ok", 
  -        "echo-interval": 0, 
  -        "local": "2001:db8:1::1", 
  -        "multihop": true, 
  -        "peer": "2001:db8:4::1", 
  -        "receive-interval": 300, 
  -        "remote-diagnostic": "ok", 
  -        "remote-echo-interval": 50, 
  -        "remote-receive-interval": 300, 
  -        "remote-transmit-interval": 300, 
  -        "status": "up", 
  -        "transmit-interval": 300
  -    }
  -]
  +[])
r1: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    857 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      2 *        264
    Connected                     :      4 *         48
    Link List                     :     34 *         40
    Link Node                     :    466 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      7 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      6 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      2 *        264
    Prefix                        :      4 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     13 *         56
    Route node                    :     19 * (variably sized)
    Thread                        :     24 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      2 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      2 *        360
    Route Entry                   :     11 *         80
    RIB destination               :     11 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      6 *         80
    Nexthop Group Connected       :      4 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      4 *         16
    Nexthop tracking object       :      2 *        232
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
r3: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    857 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      2 *        264
    Connected                     :      3 *         48
    Link List                     :     34 *         40
    Link Node                     :    465 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      6 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      6 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      2 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     13 *         56
    Route node                    :     19 * (variably sized)
    Thread                        :     24 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      2 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      2 *        360
    Route Entry                   :      9 *         80
    RIB destination               :     11 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      6 *         80
    Nexthop Group Connected       :      4 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      4 *         16
    Nexthop tracking object       :      2 *        232
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
2020-01-09 01:15:04,808 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_memory_leak": 
r1: zebra has memory leaks:

r3: zebra has memory leaks:

r4: Daemon zebra not running

From frr r4 zebra log file:
2020/01/09 01:15:12 ZEBRA: [EC 100663311] &(struct stream): 0x55a8fad6c480, size: 94184251251136, getp: 545, endp: 140017259293856


r1: Daemon zebra not running

From frr r1 zebra log file:
2020/01/09 01:15:12 ZEBRA: [EC 100663311] &(struct stream): 0x556120ff74e0, size: 93875653806416, getp: 465, endp: 140248993295520


r2: Daemon zebra not running

From frr r2 zebra log file:
2020/01/09 01:15:12 ZEBRA: [EC 100663311] &(struct stream): 0x5645b7c14c60, size: 0, getp: 279457, endp: 0


2020-01-09 01:15:14,221 ERROR: assert failed at "test_bfd_topo1/test_bfd_connection": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running

r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running
2020-01-09 01:15:25,418 ERROR: assert failed at "test_bfd_topo2/test_protocols_convergence": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running

r2: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    858 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      4 *        264
    Connected                     :      5 *         48
    Link List                     :     41 *         40
    Link Node                     :    467 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      3 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      3 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      4 *        264
    Prefix                        :      5 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :     12 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     15 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     26 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     24 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      4 *        360
    Route Entry                   :      6 *         80
    RIB destination               :     10 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      3 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      4 *         16
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
2020-01-09 01:15:25,657 ERROR: assert failed at "test_bfd_topo2/test_memory_leak": 
r2: zebra has memory leaks:

r4: Daemon zebra not running

From frr r4 zebra log file:
ZEBRA: Received signal 11 at 1578532532 (si_addr 0x555ca146f008, PC 0x7f0a47e097d0); aborting...
Program counter: /lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x40)[0x7f0a47e097d0]
Backtrace for 11 stack frames:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x60)[0x7f0a487ddbc0]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0x10c)[0x7f0a487de01c]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x6ca24)[0x7f0a487fea24]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f0a48172890]
/lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x40)[0x7f0a47e097d0]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(qfree+0x2c)[0x7f0a487e073c]
/usr/lib/frr/zebra(+0x778b4)[0x555ca01b48b4]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x56)[0x7f0a4880c236]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x3db98)[0x7f0a487cfb98]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f0a481676db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f0a47e9088f]
in thread zserv_write scheduled from zebra/zserv.c:466

r1: Daemon zebra not running

From frr r1 zebra log file:
2020/01/09 01:15:32 ZEBRA: [EC 100663311] &(struct stream): 0x55ca8aa1d480, size: 0, getp: 140132837416096, endp: 0


r2: Daemon zebra not running
r3: Daemon zebra not running
2020-01-09 01:15:35,428 ERROR: assert failed at "test_bfd_vrf_topo1/test_bfd_connection": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running

r4: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    861 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      3 *        264
    Connected                     :      3 *         48
    Link List                     :     37 *         40
    Link Node                     :    468 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      4 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :      4 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      1 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     26 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     28 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     26 *         72
    Typed-hash bucket             :     14 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      3 *        360
    Route Entry                   :      8 *         80
    RIB destination               :     15 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      4 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      1 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
r2: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    870 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      5 *        264
    Connected                     :      7 *         48
    Link List                     :     53 *         40
    Link Node                     :    478 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :     11 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :     10 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      3 *        264
    Prefix                        :      7 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :     25 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     28 *         56
    Route node                    :     39 * (variably sized)
    Thread                        :     30 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :     16 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      5 *        360
    Route Entry                   :     18 *         80
    RIB destination               :     19 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :     10 *         80
    Nexthop Group Connected       :      7 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      3 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
r3: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    860 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      3 *        264
    Connected                     :      3 *         48
    Link List                     :     37 *         40
    Link Node                     :    468 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      4 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :      4 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      1 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     26 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     28 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :     14 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      3 *        360
    Route Entry                   :      8 *         80
    RIB destination               :     15 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      4 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      1 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
2020-01-09 01:15:36,019 ERROR: assert failed at "test_bfd_vrf_topo1/test_memory_leak": 
r4: zebra has memory leaks:

r2: zebra has memory leaks:

r3: zebra has memory leaks:

*** defaultIntf: warning: r1 has no interfaces
2020-01-09 01:52:56,063 ERROR: '_bgp_has_routes' failed after 38.44 seconds
2020-01-09 02:00:35,577 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp community-list standard ANY permit 0:-1 



2020-01-09 02:00:35,693 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp community-list standard ANY permit 0:65536 



2020-01-09 02:00:35,814 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp large-community-list standard ANY permit 0:4294967296 



2020-01-09 02:00:35,953 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp large-community-list standard ANY permit 0:-1:1 



2020-01-09 02:02:37,444 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1804/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: line 2: % Command incomplete[4]: bgp large-community-list standard Test1 permit  

see full log at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOU1804/ErrorLog/log_topotests.txt

CentOS 7 rpm pkg check: Failed (click for details)

CentOS 7 rpm pkg check: Unknown Log URL: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/CENTOS7RPM/ErrorLog/log_package_start.txt CentOS 7 rpm pkg check: No useful log found

Ubuntu 16.04 deb pkg check: Failed (click for details)

Ubuntu 16.04 deb pkg check: Unknown Log URL: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/U1604DEBTEST/ErrorLog/log_package_start.txt Ubuntu 16.04 deb pkg check: No useful log found

Topotest tests on Ubuntu 16.04 i386: Failed (click for details)

Topology Test Results are at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOI386-10357/test

Topology Tests failed for Topotest tests on Ubuntu 16.04 i386
see full log at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOI386/ErrorLog/log_topotests.txt

Fedora 29 rpm pkg check: Failed (click for details)

Fedora 29 rpm pkg check: Unknown Log URL: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/F29RPMCHECK/ErrorLog/log_package_start.txt Fedora 29 rpm pkg check: No useful log found

Topology tests on Ubuntu 16.04 amd64: Failed (click for details)

Topology Test Results are at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOU1604-10357/test

Topology Tests failed for Topology tests on Ubuntu 16.04 amd64:

r3: Daemon zebra not running

From frr r3 zebra log file:
ZEBRA: Received signal 11 at 1578532473 (si_addr 0x55f18c87c008, PC 0x7f17ce9a4098); aborting...
Program counter: /lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7f17ce9a4098]
Backtrace for 11 stack frames:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x67)[0x7f17cf364457]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0x113)[0x7f17cf3648b3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x6c495)[0x7f17cf385495]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f17cecfa390]
/lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7f17ce9a4098]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(qfree+0x2c)[0x7f17cf36722c]
/usr/lib/frr/zebra(+0x770d3)[0x55f18af2e0d3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x60)[0x7f17cf392d60]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x3d5c8)[0x7f17cf3565c8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f17cecf06ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f17cea2641d]
in thread zserv_write scheduled from zebra/zserv.c:466

2020-01-08 17:14:33,323 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_protocols_convergence": r3: Daemon zebra not running

2020-01-08 17:14:38,657 ERROR: 'router_json_cmp' failed after 5.14 seconds
2020-01-08 17:14:38,658 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_bfd_comes_back_again": "r1" JSON output mismatches
assert json too few items (have 0, expected 1:
   --- Expected value
  +++ Current value
  @@ -1,16 +1 @@
  -[
  -    {
  -        "diagnostic": "ok", 
  -        "echo-interval": 0, 
  -        "local": "2001:db8:1::1", 
  -        "multihop": true, 
  -        "peer": "2001:db8:4::1", 
  -        "receive-interval": 300, 
  -        "remote-diagnostic": "ok", 
  -        "remote-echo-interval": 50, 
  -        "remote-receive-interval": 300, 
  -        "remote-transmit-interval": 300, 
  -        "status": "up", 
  -        "transmit-interval": 300
  -    }
  -]
  +[])
r1: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    857 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      2 *        264
    Connected                     :      4 *         48
    Link List                     :     34 *         40
    Link Node                     :    466 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      7 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      6 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      2 *        264
    Prefix                        :      4 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     13 *         56
    Route node                    :     19 * (variably sized)
    Thread                        :     24 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      2 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      2 *        360
    Route Entry                   :     11 *         80
    RIB destination               :     11 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      6 *         80
    Nexthop Group Connected       :      4 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      4 *         16
    Nexthop tracking object       :      2 *        232
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
r3: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    857 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      2 *        264
    Connected                     :      3 *         48
    Link List                     :     34 *         40
    Link Node                     :    465 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      6 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      6 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      2 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     13 *         56
    Route node                    :     19 * (variably sized)
    Thread                        :     24 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      2 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      2 *        360
    Route Entry                   :      9 *         80
    RIB destination               :     11 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      6 *         80
    Nexthop Group Connected       :      4 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      4 *         16
    Nexthop tracking object       :      2 *        232
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
2020-01-08 17:14:38,801 ERROR: assert failed at "test_bfd_bgp_cbit_topo3/test_memory_leak": 
r1: zebra has memory leaks:

r3: zebra has memory leaks:

r4: Daemon zebra not running

From frr r4 zebra log file:
2020/01/08 17:14:46 ZEBRA: [EC 100663311] &(struct stream): 0x562e30a1ec60, size: 94756384407920, getp: 545, endp: 140036856114040


r1: Daemon zebra not running

From frr r1 zebra log file:
2020/01/08 17:14:46 ZEBRA: [EC 100663311] &(struct stream): 0x5557b73c0c40, size: 93835224687264, getp: 401, endp: 140358527724408


r2: Daemon zebra not running

From frr r2 zebra log file:
2020/01/08 17:14:46 ZEBRA: [EC 100663311] &(struct stream): 0x5614c506ea10, size: 94647204753488, getp: 2497, endp: 139707952114552


r3: Daemon zebra not running

From frr r3 zebra log file:
2020/01/08 17:14:48 ZEBRA: [EC 100663311] &(struct stream): 0x556b068fd3a0, size: 93918159752976, getp: 85089, endp: 93918159934016


2020-01-08 17:14:48,339 ERROR: assert failed at "test_bfd_topo1/test_bfd_connection": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running

r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running
2020-01-08 17:14:59,068 ERROR: assert failed at "test_bfd_topo2/test_protocols_convergence": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running

r1: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      2 *       4120
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    855 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      2 *        264
    Connected                     :      3 *         48
r2: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3890 *         72
    Command Token Text            :   2837 * (variably sized)
    Command Token Help            :   2837 * (variably sized)
    Command Argument Name         :    894 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4556 *         32
    Hash                          :    856 * (variably sized)
    Hash Bucket                   :    861 *         32
    Hash Index                    :    428 * (variably sized)
    Hook entry                    :     19 *         48
    Interface                     :      4 *        264
    Connected                     :      5 *         48
    Link List                     :     41 *         40
    Link Node                     :    467 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      3 *        120
    NetNS Context                 :      2 * (variably sized)
    NetNS Name                    :      1 *         18
    Nexthop Group                 :      3 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      4 *        264
    Prefix                        :      5 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :     11 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     15 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     26 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     24 *         72
    Typed-hash bucket             :      8 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9167 *         16
    Vector index                  :   9167 * (variably sized)
    VRF                           :      1 *        200
    VRF bit-map                   :      7 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      4 *        360
    Route Entry                   :      6 *         80
    RIB destination               :     10 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      3 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :      5 * (variably sized)
    PTM BFD process registration table.:      2 *         32
    RIB table info                :      4 *         16
    ZEBRA VRF                     :      1 *       4744

  ## showing active allocations in memory group Table Manager
2020-01-08 17:14:59,282 ERROR: assert failed at "test_bfd_topo2/test_memory_leak": 
r1: zebra has memory leaks:

r2: zebra has memory leaks:

r4: Daemon zebra not running

From frr r4 zebra log file:
ZEBRA: Received signal 11 at 1578532506 (si_addr 0x55b2076fd008, PC 0x7fb2491a4098); aborting...
Program counter: /lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7fb2491a4098]
Backtrace for 11 stack frames:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x67)[0x7fb249b64457]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0x113)[0x7fb249b648b3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x6c495)[0x7fb249b85495]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fb2494fa390]
/lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7fb2491a4098]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(qfree+0x2c)[0x7fb249b6722c]
/usr/lib/frr/zebra(+0x770d3)[0x55b2051950d3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x60)[0x7fb249b92d60]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x3d5c8)[0x7fb249b565c8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fb2494f06ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fb24922641d]
in thread zserv_write scheduled from zebra/zserv.c:466

r1: Daemon zebra not running

From frr r1 zebra log file:
ZEBRA: Received signal 11 at 1578532506 (si_addr 0x561d419c7008, PC 0x7fe6de44f098); aborting...
Program counter: /lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7fe6de44f098]
Backtrace for 11 stack frames:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x67)[0x7fe6dee0f457]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0x113)[0x7fe6dee0f8b3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x6c495)[0x7fe6dee30495]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fe6de7a5390]
/lib/x86_64-linux-gnu/libc.so.6(malloc_usable_size+0x28)[0x7fe6de44f098]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(qfree+0x2c)[0x7fe6dee1222c]
/usr/lib/frr/zebra(+0x770d3)[0x561d3f2c70d3]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x60)[0x7fe6dee3dd60]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x3d5c8)[0x7fe6dee015c8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fe6de79b6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe6de4d141d]
in thread zserv_write scheduled from zebra/zserv.c:466

r2: Daemon zebra not running
r3: Daemon zebra not running
2020-01-08 17:15:09,073 ERROR: assert failed at "test_bfd_vrf_topo1/test_bfd_connection": r4: Daemon zebra not running
r1: Daemon zebra not running
r2: Daemon zebra not running
r3: Daemon zebra not running

r4: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    861 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      3 *        264
    Connected                     :      3 *         48
    Link List                     :     37 *         40
    Link Node                     :    468 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      4 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :      4 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      1 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     26 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     28 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     26 *         72
    Typed-hash bucket             :     14 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      3 *        360
    Route Entry                   :      8 *         80
    RIB destination               :     15 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      4 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      1 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
r1: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      4 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    861 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      3 *        264
    Connected                     :      3 *         48
    Link List                     :     37 *         40
    Link Node                     :    468 *         24
    Logging                       :      2 * (variably sized)
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      4 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :      4 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      1 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     26 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     28 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     26 *         72
    Typed-hash bucket             :     14 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      3 *        360
    Route Entry                   :      8 *         80
    RIB destination               :     15 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      4 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      1 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
r2: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    870 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      5 *        264
    Connected                     :      7 *         48
    Link List                     :     53 *         40
    Link Node                     :    478 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :     11 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :     10 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      3 *        264
    Prefix                        :      7 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     28 *         56
    Route node                    :     39 * (variably sized)
    Thread                        :     30 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :     16 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      5 *        360
    Route Entry                   :     18 *         80
    RIB destination               :     19 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :     10 *         80
    Nexthop Group Connected       :      7 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      3 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
r3: zebra has memory leaks:

  ## showing active allocations in memory group libfrr
    Buffer                        :      4 *         24
    Buffer data                   :      1 *       4120
    Host config                   :      3 * (variably sized)
    Command Tokens                :   3910 *         72
    Command Token Text            :   2851 * (variably sized)
    Command Token Help            :   2851 * (variably sized)
    Command Argument Name         :    900 * (variably sized)
    RCU thread                    :      5 *        128
    FRR POSIX Thread              :     10 * (variably sized)
    POSIX sync primitives         :     10 * (variably sized)
    Graph                         :     25 *          8
    Graph Node                    :   4580 *         32
    Hash                          :    864 * (variably sized)
    Hash Bucket                   :    860 *         32
    Hash Index                    :    432 * (variably sized)
    Hook entry                    :     20 *         48
    Interface                     :      3 *        264
    Connected                     :      3 *         48
    Link List                     :     37 *         40
    Link Node                     :    468 *         24
    Logging                       :      1 *         72
    Temporary memory              :      4 * (variably sized)
    Nexthop                       :      4 *        120
    NetNS Context                 :      4 * (variably sized)
    NetNS Name                    :      2 * (variably sized)
    Nexthop Group                 :      4 *          8
    Northbound Node               :      5 *        400
    Northbound Configuration      :      2 *         16
    Northbound Configuration Entry:      1 *        264
    Prefix                        :      3 *         48
    Privilege information         :      4 * (variably sized)
    Stream                        :      9 * (variably sized)
    Stream FIFO                   :     10 *         64
    Route table                   :     26 *         56
    Route node                    :     21 * (variably sized)
    Thread                        :     28 *        168
    Thread master                 :     23 * (variably sized)
    Thread Poll Info              :     12 *     131072
    Thread stats                  :     25 *         72
    Typed-hash bucket             :     14 * (variably sized)
    Typed-heap array              :      1 *        576
    Vector                        :   9215 *         16
    Vector index                  :   9215 * (variably sized)
    VRF                           :      2 *        200
    VRF bit-map                   :      4 *          8
    Work queue                    :      3 * (variably sized)
    Work queue name string        :      2 * (variably sized)
    YANG module                   :      1 *         48

  ## showing active allocations in memory group Label Manager

  ## showing active allocations in memory group zebra
    Zebra Interface Information   :      3 *        360
    Route Entry                   :      8 *         80
    RIB destination               :     15 *         88
    Zebra DPlane Provider         :      1 *        232
    Nexthop Group Entry           :      4 *         80
    Nexthop Group Connected       :      3 *         40
    Zebra Name Space              :     11 * (variably sized)
    PTM BFD process registration table.:      1 *         32
    RIB table info                :      9 *         16
    Nexthop tracking object       :      1 *        232
    ZEBRA VRF                     :      2 *       4744
    Other Table                   :      1 *         56

  ## showing active allocations in memory group Table Manager
2020-01-08 17:15:09,628 ERROR: assert failed at "test_bfd_vrf_topo1/test_memory_leak": 
r4: zebra has memory leaks:

r1: zebra has memory leaks:

r2: zebra has memory leaks:

r3: zebra has memory leaks:

*** defaultIntf: warning: r1 has no interfaces
2020-01-08 17:53:18,101 ERROR: '_bgp_has_routes' failed after 39.17 seconds
2020-01-08 18:00:01,677 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp community-list standard ANY permit 0:-1 



2020-01-08 18:00:01,807 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp community-list standard ANY permit 0:65536 



2020-01-08 18:00:01,945 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp large-community-list standard ANY permit 0:4294967296 



2020-01-08 18:00:02,090 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: % Malformed community-list value
line 2: Failure to communicate[13] to bgpd, line: bgp large-community-list standard ANY permit 0:-1:1 



2020-01-08 18:02:00,801 ERROR: Traceback (most recent call last):
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 1514, in create_bgp_community_lists
    build=build)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 236, in create_common_configuration
    load_config_to_router(tgen, router)
  File "/root/bamboo-agent-home/xml-data/build-dir/FRR-FRRPULLREQ-TOPOU1604/topotests/lib/common_config.py", line 389, in load_config_to_router
    raise InvalidCLIError("%s" % output)
InvalidCLIError: line 2: % Command incomplete[4]: bgp large-community-list standard Test1 permit  

see full log at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOU1604/ErrorLog/log_topotests.txt

Successful on other platforms

• IPv6 protocols on Ubuntu 14.04
• Debian 10 deb pkg check
• Ubuntu 12.04 deb pkg check
• Addresssanitizer topotest
• Debian 8 deb pkg check
• IPv4 protocols on Ubuntu 14.04
• IPv4 ldp protocol on Ubuntu 16.04
• Ubuntu 14.04 deb pkg check
• Debian 9 deb pkg check
• Static analyzer (clang)

Topology Tests memory analysis: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOU1804/MemoryLeaks/
Topology Tests memory analysis: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOI386/MemoryLeaks/
Topology Tests memory analysis: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/TOPOU1604/MemoryLeaks/

Warnings Generated during build:

Debian 10 amd64 build: Successful with additional warnings

Debian Package lintian failed for Debian 10 amd64 build:
(see full package build log at https://ci1.netdef.org/browse/FRR-FRRPULLREQ-10357/artifact/DEB10BUILD/ErrorLog/log_lintian.txt)

W: frr source: pkg-js-tools-test-is-missing
W: frr source: pkg-js-tools-test-is-missing
W: frr-rpki-rtrlib: changelog-file-missing-explicit-entry 6.0-2 -> 7.3-dev-20200109-00-gd13785b01-0 (missing) -> 7.3-dev-20200109-00-gd13785b01-0~deb10u1
W: frr-doc: changelog-file-missing-explicit-entry 6.0-2 -> 7.3-dev-20200109-00-gd13785b01-0 (missing) -> 7.3-dev-20200109-00-gd13785b01-0~deb10u1
W: frr: changelog-file-missing-explicit-entry 6.0-2 -> 7.3-dev-20200109-00-gd13785b01-0 (missing) -> 7.3-dev-20200109-00-gd13785b01-0~deb10u1
W: frr-snmp: changelog-file-missing-explicit-entry 6.0-2 -> 7.3-dev-20200109-00-gd13785b01-0 (missing) -> 7.3-dev-20200109-00-gd13785b01-0~deb10u1
W: frr-pythontools: changelog-file-missing-explicit-entry 6.0-2 -> 7.3-dev-20200109-00-gd13785b01-0 (missing) -> 7.3-dev-20200109-00-gd13785b01-0~deb10u1

<TITLE>clang_check</TITLE>

clang_check