Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graphd may coredump when enable_space_level_metrics = true #5555

Closed
flymysql opened this issue May 17, 2023 · 2 comments
Closed

Graphd may coredump when enable_space_level_metrics = true #5555

flymysql opened this issue May 17, 2023 · 2 comments
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Milestone

Comments

@flymysql
Copy link
Contributor

Please check the FAQ documentation before raising an issue

Describe the bug (required)
my nebula version is v3.2.1

*** Aborted at 1684118566 (Unix time, try 'date -d @1684118566') ***
*** Signal 11 (SIGSEGV) (0x58) received by PID 3399 (pthread TID 0x7fc8562fc700) (linux TID 3442) (code: address not mapped to object), stack trace: ***
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x55c64431d5d1]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x25)[0x55c644313195]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0x23f6ecb)[0x55c644310ecb]
/lib64/libpthread.so.0(+0xf62f)[0x7fc865b8662f]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN6nebula5stats12StatsManager17counterWithLabelsERKNS0_9CounterIdERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESC_ESaISD_EE+0x2bc)[0x55c643d5d63c]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZNK6nebula5graph13QueryInstance17addSlowQueryStatsEmRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1ba)[0x55c642e4ca1a]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN6nebula5graph13QueryInstance8onFinishEv+0x118)[0x55c642e4d838]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0xf346aa)[0x55c642e4e6aa]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0x23680db)[0x55c6442820db]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN5folly7futures6detail8CoreBase10doCallbackEONS_8Executor9KeepAliveIS3_EENS1_5StateE+0x399)[0x55c644282a39]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZNK6nebula5graph28AsyncMsgNotifyBasedScheduler8notifyOKERSt6vectorIN5folly7PromiseINS_6StatusEEESaIS6_EE+0x91)[0x55c64317adf1]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0x12610c3)[0x55c64317b0c3]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0x12612b4)[0x55c64317b2b4]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(+0x23680db)[0x55c6442820db]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN6apache6thrift11concurrency14FunctionRunner3runEv+0x7f)[0x55c64404618f]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN6apache6thrift11concurrency13ThreadManager4Impl6Worker3runEv+0x204)[0x55c6441824e4]
/opt/huawei/l00579656/online_server/master/nebula2/bin/nebula-graphd(_ZN6apache6thrift11concurrency13PthreadThread10threadMainEPv+0xdc)[0x55c6441861ec]
/lib64/libpthread.so.0(+0x7ea4)[0x7fc865b7eea4]
/lib64/libc.so.6(clone+0x6c)[0x7fc8658a7b0c]
(safe mode, symbolizer not available)

Your Environments (required)

  • OS: uname -a
Linux dggphispre07207 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Compiler: g++ --version or clang++ --version
g++ (Nebula Graph Build) 10.1.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  • CPU: lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6161 CPU @ 2.20GHz
Stepping:              4
CPU MHz:               2200.000
BogoMIPS:              4400.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              30976K
NUMA node0 CPU(s):     0-7
NUMA node1 CPU(s):     8-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat md_clear spec_ctrl intel_stibp flush_l1d
You have new mail in /var/spool/mail/tysearch
  • Commit id (e.g. a3ffc7d8)
bb2e684
v3.2.1

How To Reproduce(required)

Steps to reproduce the behavior:

  1. Step 1
    enable_space_level_metrics = true
  2. Step 2
    normal run write request

Expected behavior

Additional context

@flymysql flymysql added the type/bug Type: something is unexpected label May 17, 2023
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels May 17, 2023
@tangyuanzhang
Copy link
Contributor

stats的实现里面,有几个map,但是只在写的时候加锁了,读没有加锁,高并发的请求可能会core,猜测可能是这个导致的,可以重新编译个RelWithDebInfo 来复现,这个coredump文件被优化了,不容易看出问题

@Sophie-Xie
Copy link
Contributor

fixed in #5621

@Sophie-Xie Sophie-Xie added this to the v3.6.0 milestone Jul 12, 2023
@github-actions github-actions bot added the process/fixed Process of bug label Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

3 participants