Crash related to broker python bindings when running zeekctl

[timwoj]

I ran into this while setting up a Docker configuration to run long-term benchmarks of zeek. Python crashed when running zeekctl deploy and while minimizing zeekctl to a smaller script, it looked like it was the broker bindings causing it to crash. Adding --disable-python to my zeek build ultimately resolved it but I'm reporting this anyways.

Python backtrace from gdb: backtrace.txt
Python verbose output running minimized script: python_verbose.txt
Minimized script: test.py.txt
Dockerfile used for testing: Dockerfile.txt

[rsmmr]

@Neverlord any idea?

[Neverlord]

Not yet. I couldn't spot something obvious in the output other than that it seems to die in Broker's configuration class.

[timwoj]

Here's another thread with people running into the same: https://lists.zeek.org/archives/list/[email protected]/thread/BCAQ4G3HZ244W4EGRVPK4KHER67KCNVH/

Seems like CentOS 8.4 can trigger it pretty readily, where as 8.3 is hit or miss. The current Docker image for CentOS is still 8.3.

[timwoj]

A slightly better stacktrace:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff28a9699 in broker::topic::topic<char const (&) [4], void> (x=..., 
    this=0x7ffff2cb7d20 <broker::topics::reserved>) at /usr/include/c++/8/bits/move.h:74
#2  __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
    at ../auxil/broker/include/broker/topic.hh:87
#3  _GLOBAL__sub_I_configuration.cc(void) () at ../auxil/broker/src/configuration.cc:219

It looks like it might be something with GCC too. It builds and runs fine with Clang 11 on CentOS 8.4, but the default version of GCC (8.4.1) crashes.

[grigorescu]

Tried using faulthandler:

[root@23571ab70bdc /]# python3 -q -X faulthandler
>>> import sys
>>> sys.path.insert(0, "/opt/zeek_from_src/lib64/zeek/python")
>>> sys.path.insert(0, "/opt/zeek_from_src/lib64/zeek/python/zeekctl")
>>> from ZeekControl.zeekctl import ZeekCtl
Fatal Python error: Segmentation fault

Current thread 0x00007f6db6bbab80 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 658 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/opt/zeek_from_src/lib64/zeek/python/broker/__init__.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "/opt/zeek_from_src/lib64/zeek/python/zeekctl/ZeekControl/events.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/opt/zeek_from_src/lib64/zeek/python/zeekctl/ZeekControl/control.py", line 11 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/opt/zeek_from_src/lib64/zeek/python/zeekctl/ZeekControl/zeekctl.py", line 12 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<stdin>", line 1 in <module>
Segmentation fault

[grigorescu]

And one more, just for broker:

[root@23571ab70bdc /]# python3 -q -X faulthandler
>>> import sys
>>> sys.path.insert(0, "/opt/zeek_from_src/lib64/zeek/python")
>>> import broker
Fatal Python error: Segmentation fault

Current thread 0x00007fe5a1eafb80 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 658 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1023 in _handle_fromlist
  File "/opt/zeek_from_src/lib64/zeek/python/broker/__init__.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<stdin>", line 1 in <module>
Segmentation fault

[Neverlord]

@grigorescu @timwoj thanks a lot for the input! I can reproduce this on CentOS, but investigating wasn't so straightforward since it won't reproduce with ASAN builds. Judging from traces you've provided (in particular seeing this fail in __static_initialization_and_destruction_0) I think this ultimately boils down to static initialization order fiasco with the topic singletons living in broker::topics.

After patching the singletons out (see #191 and zeek/zeek#1646), it no longer crashes for me (crashes every time when using the master branch).