Issue3373 storage exit crash #3553
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cangfengzhs
force-pushed
the
issue3373-storage-exit-crash
branch
from
46dcfa5
to
cbc618f
Compare
critical27
reviewed
Dec 29, 2021
critical27
requested review from
liuyu85cn,
CPWstatic,
wenhaocs and
yixinglu
cangfengzhs
force-pushed
the
issue3373-storage-exit-crash
branch
from
1f53617
to
a5f3f65
Compare
cangfengzhs
force-pushed
the
issue3373-storage-exit-crash
branch
from
bf32da3
to
7081904
Compare
critical27
approved these changes
Jan 7, 2022
CPWstatic
approved these changes
Jan 7, 2022
Sophie-Xie
pushed a commit
that referenced
this pull request
Jan 10, 2022
* use rcu replace thread local fix storage exit crash format address some comment * fix bug * fix bug
critical27
added a commit
that referenced
this pull request
Jan 10, 2022
* Fix typos (#3615) Co-authored-by: kyle.cao <[email protected]> * fix fetch edges tostring (#3613) Co-authored-by: Sophie <[email protected]> Co-authored-by: Yichen Wang <[email protected]> * fix create space assign offline host (#3583) * fix create space * fix test case Co-authored-by: Harris.Chu <[email protected]> * Disable ARM version docker image since related third party not ready (#3618) * Unify raft error code (#3620) * Meta upgrader v3 (#3540) * Replace group when create space * Support white list * fix test case * support zone operations * fix * Support meta upgrade v3 * add more check about parse host result (#3628) * Ut fix (#3611) * Enable ut and fix chaindelete * Add mock server default worker * fix service crash (#3616) * Cleanup branch param in package script (#3622) * fix crash when the expression exceed the depth (#3606) * Enhance login password check (#3629) * fix_batch_insert_problem (#3627) * filter data before batch insert * add test cases * add more testcase * add notifyStop() for metaClient (#3621) * add notifyStop() for metaClient * do clean * Fix removeSession() (#3651) Co-authored-by: Yee <[email protected]> * Issue3373 storage exit crash (#3553) * use rcu replace thread local fix storage exit crash format address some comment * fix bug * fix bug * Fix coalesce bug (#3653) * fix coalesce * fix test * add test * add tck * fix * fix * fix * delete double check agg in where clause (#3647) Co-authored-by: Yee <[email protected]> Co-authored-by: cpw <[email protected]> * fix meta crash after create space (#3660) Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: kyle.cao <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: Yee <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: cpw <[email protected]>
yixinglu
pushed a commit
to yixinglu/nebula
that referenced
this pull request
Mar 21, 2022
* use rcu replace thread local fix storage exit crash format address some comment * fix bug * fix bug fix bug fix bug Co-authored-by: hs.zhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What does this PR do?
Use RCU replace ThreadLocal in MetaClient
Which issue(s)/PR(s) this PR relates to?
#3373
#3497
Special notes for your reviewer, ex. impact of this fix, etc:
At the beginning, we found that storage would crash after running for a long time (a large number of insert edge operations were performed at the same time). At the same time, Storage's memory usage will be very high. So we guess that there is a memory leak after the system OOM. However, it was later discovered that this is not the problem. Even if the Storage does not have OOM, it will crash when it is stopped. All coredump stacks destruct a static thread variable when the thread exits. This is a variable of type folly::SingletonThreadLocal introduced in MetaClient.
At the same time, in another scenario, if compaction is triggered when storage is started, it will crash directly, and the coredump stack and stop will be the same.
After a long time of investigation, we did not find the specific cause of this problem, but we found that this was a problem that only appeared after the introduction of folly::SingletonThreadLocal, so we chose to deprecate folly::SingletonThreadLocal and replace it with RCU it.
After using RCU, there is indeed no crash. I am not sure whether it was really fixed or just because the probability of crash has decreased and I did not find it.
In addition, the performance of using RCU should also be better than the performance of ThreadLocal, because no read-write lock means no blocking
Additional context/ Design document:
Checklist:
Release notes:
Please confirm whether to be reflected in release notes and how to describe: