nebula-storaged core dump

[zedzhu]

Describe the bug (required)

nebula-storaged core dumped while I import a large dataset into a single nebula node. Below are logs and core stack.

import logs:
1729 2021/11/08 18:07:31 [INFO] statsmgr.go:62: Tick: Time(8600.00s), Finished(931066660), Failed(0), Read Failed(0), Latency AVG(4296us), Batches Req AVG(4560us), Rows AVG(108263.56/s)
1730 2021/11/08 18:07:32 [ERROR] handler.go:63: Client 2 fail to execute: INSERT VERTEX word_id(name,type) VALUES hash("过墙钻头zhaozi"): ("过墙钻头zhaozi","pfx"),...;, ErrMsg: Storage Error: part: 4, error: E_RPC_FAILURE(-3)., ErrCode: -1005

core stack:
(gdb) bt
#0 0x00007f5d4609a337 in raise () from /lib64/libc.so.6
#1 0x00007f5d4609ba28 in abort () from /lib64/libc.so.6
#2 0x000000000200f7ad in google::LogMessage::Fail() ()
#3 0x00000000020119b5 in google::LogMessage::SendToLog() ()
#4 0x000000000200f2cd in google::LogMessage::Flush() ()
#5 0x0000000002012359 in google::LogMessageFatal::~LogMessageFatal() ()
#6 0x00000000017678f1 in nebula::raftex::RaftPart::processAppendLogResponses(std::vector<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse>, std::allocator<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse> > > const&, folly::EventBase*, nebula::raftex::AppendLogsIterator, long, long, long, long, long, std::vector<std::shared_ptrnebula::raftex::Host, std::allocator<std::shared_ptrnebula::raftex::Host > >) ()
#7 0x0000000001767abf in nebula::raftex::RaftPart::replicateLogs(folly::EventBase*, nebula::raftex::AppendLogsIterator, long, long, long, long, long)::{lambda(folly::Try<std::vector<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse>, std::allocator<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse> > > >&&)#3}::operator()(folly::Try<std::vector<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse>, std::allocator<std::pair<unsigned long, nebula::raftex::cpp2::AppendLogResponse> > > >&&) ()
#8 0x0000000001767f5f in _ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEE7callBigIZNS4_4CoreISt6vectorISt4pairImN6nebula6raftex4cpp217AppendLogResponseEESaISN_EEE11setCallbackIZNS4_10FutureBaseISP_E18thenImplementationIZNOS_6FutureISP_E7thenTryIZNSK_8RaftPart13replicateLogsEPNS_9EventBaseENSK_18AppendLogsIteratorElllllEUlONS_3TryISP_EEE1_EENSV_INS4_17tryCallableResultISP_T_NSt9enable_ifIX14is_invocable_vIS17_S14_EEvE4typeEE10value_typeEEEOS17_EUlSA_S14_E_NS4_25tryExecutorCallableResultISP_S1F_vEEEENS18_IXntsrNT0_13ReturnsFutureE5valueENS1I_6ReturnEE4typeES1E_S1I_NS4_18InlineContinuationEEUlSA_S14_E_EEvS1E_OSt10shared_ptrINS_14RequestContextEES1N_EUlS6_SA_SC_E_EEvS6_SA_SC_RNS1_4DataE ()
#9 0x0000000001f2eefc in ?? ()
#10 0x0000000001b92ed9 in apache::thrift::concurrency::FunctionRunner::run() ()
#11 0x0000000001cbea62 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#12 0x0000000001cc2a28 in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#13 0x00007f5d46439e65 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f5d4616288d in clone () from /lib64/libc.so.6

Your Environments (required)

• OS: CentOS Linux release 7.2.1511 (Core)
• Compiler: g++ (Nebula Graph Build) 7.5.0
• CPU: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
  Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
  CPU(s): 16
  On-line CPU(s) list: 0-15
  Thread(s) per core: 2
  Core(s) per socket: 8

nebula version: nebula-2.6.0

[critical27]

Usually will core in this step because of we write data to rocksdb failed. One of the most common scene is that the disk is full.