Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metad crashing after clean install #4957

Closed
TheDome0 opened this issue Nov 29, 2022 · 4 comments
Closed

metad crashing after clean install #4957

TheDome0 opened this issue Nov 29, 2022 · 4 comments
Assignees
Labels
affects/none PR/issue: this bug affects none version. need info Solution: need more information (ex. can't reproduce) process/done Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Milestone

Comments

@TheDome0
Copy link

TheDome0 commented Nov 29, 2022

Describe the bug (required)
metad keeps crashing immediately after startup

*** Aborted at 1669742004 (Unix time, try 'date -d @1669742004') ***
*** Signal 6 (SIGABRT) (0x111f6) received by PID 70134 (pthread TID 0x7f77691ed080) (linux TID 70134) (maybe from PID 70134, UID 0) (code: -6), stack trace: ***
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x253ace1]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x26)[0x25321e6]
/usr/local/nebula/bin/nebula-metad[0x2530177]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1441f)[0x7f77693f741f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f776923400b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12a)[0x7f7769213858]
/usr/local/nebula/bin/nebula-metad[0x10f770e]
/usr/local/nebula/bin/nebula-metad[0x25636c4]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStore4stopEv+0xb2)[0x1b39f92]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStoreD1Ev+0x24)[0x1b437e4]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStoreD0Ev+0x8)[0x1b43f48]
/lib/x86_64-linux-gnu/libc.so.6(+0x468a6)[0x7f77692378a6]
/lib/x86_64-linux-gnu/libc.so.6(exit+0x1f)[0x7f7769237a5f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf9)[0x7f7769215089]
/usr/local/nebula/bin/nebula-metad(_start+0x2d)[0x1142ccd]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x253ace1]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x26)[0x25321e6]
/usr/local/nebula/bin/nebula-metad[0x2530c5c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1441f)[0x7f77693f741f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f776923400b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12a)[0x7f7769213858]
/usr/local/nebula/bin/nebula-metad[0x10f824c]
/usr/local/nebula/bin/nebula-metad[0x252cce6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x114de)[0x7f77693f44de]
/usr/local/nebula/bin/nebula-metad(_ZN5folly27tryGetCurrentAsyncStackRootEv+0x63)[0x252cdb3]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer22getAsyncStackTraceSafeEPmm+0x14)[0x253af64]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x88)[0x2532248]
/usr/local/nebula/bin/nebula-metad[0x2530177]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1441f)[0x7f77693f741f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f776923400b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12a)[0x7f7769213858]
/usr/local/nebula/bin/nebula-metad[0x10f770e]
/usr/local/nebula/bin/nebula-metad[0x25636c4]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStore4stopEv+0xb2)[0x1b39f92]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStoreD1Ev+0x24)[0x1b437e4]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStoreD0Ev+0x8)[0x1b43f48]
/lib/x86_64-linux-gnu/libc.so.6(+0x468a6)[0x7f77692378a6]
/lib/x86_64-linux-gnu/libc.so.6(exit+0x1f)[0x7f7769237a5f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf9)[0x7f7769215089]
/usr/local/nebula/bin/nebula-metad(_start+0x2d)[0x1142ccd]

Your Environments (required)

  • OS: Linux / 5.4.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux

How To Reproduce(required)
Install ubuntu2004 deb package and then start all

Additional context

storaged logs are empty - all conf files are default
graphd-stderr.log
metad-stderr.log

@TheDome0 TheDome0 added the type/bug Type: something is unexpected label Nov 29, 2022
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels Nov 29, 2022
@Sophie-Xie Sophie-Xie added this to the v3.4.0 milestone Nov 30, 2022
@wey-gu
Copy link
Contributor

wey-gu commented Nov 30, 2022

@HarrisChu HarrisChu added the need info Solution: need more information (ex. can't reproduce) label Dec 7, 2022
@critical27
Copy link
Contributor

Do you have core-dump in your environment? Would you print the stack trace by gdb? And the INFO log is appreciated as well.

@TheDome0
Copy link
Author

TheDome0 commented Dec 13, 2022

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fc5db208859 in __GI_abort () at abort.c:79
#2  0x0000000001101b03 in ?? ()
#3  0x0000000002a6c3c6 in __cxxabiv1::__terminate(void (*)()) ()
#4  0x0000000002a6c431 in std::terminate() ()
#5  0x0000000002a6c584 in __cxa_throw ()
#6  0x0000000001103ae4 in std::__throw_system_error(int) ()
#7  0x0000000002aea239 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ()
#8  0x0000000001231cb1 in nebula::meta::JobManager::init(nebula::kvstore::KVStore*) ()
#9  0x00000000011057b8 in main ()

.INFO

Log file created at: 2022/12/13 20:38:20
Running on machine: h2992802.stratoserver.net
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20221213 20:38:20.845883 128948 MetaDaemon.cpp:137] localhost = "127.0.0.1":9559
I20221213 20:38:20.855808 128948 NebulaStore.cpp:51] Start the raft service...
I20221213 20:38:20.856906 128948 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20221213 20:38:20.940384 128948 RaftexService.cpp:46] Start raft service on 9560
I20221213 20:38:20.940585 128948 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20221213 20:38:20.940698 128948 NebulaStore.cpp:92] Scan path "/usr/local/nebula/data/meta/nebula/0"
I20221213 20:38:20.940742 128948 NebulaStore.cpp:271] Init data from partManager for "127.0.0.1":9559
I20221213 20:38:20.940834 128948 NebulaStore.cpp:387] Create data space 0
I20221213 20:38:20.965900 128948 RocksEngine.cpp:97] open rocksdb on /usr/local/nebula/data/meta/nebula/0/data

I hope that's what you're asking

critical27 added a commit to critical27/nebula that referenced this issue Dec 15, 2022
@critical27
Copy link
Contributor

critical27 commented Dec 22, 2022

I guess it is probably because your thread count limit in ulimit is to small. It throws exception when creating a new thread.

@github-actions github-actions bot added the process/fixed Process of bug label Dec 22, 2022
@Hester-Gu Hester-Gu added the process/done Process of bug label Jan 13, 2023
@github-actions github-actions bot removed the process/fixed Process of bug label Jan 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/none PR/issue: this bug affects none version. need info Solution: need more information (ex. can't reproduce) process/done Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

6 participants