Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DLedger RPC Service thread stuck #282

Closed
RongtongJin opened this issue Mar 18, 2023 · 0 comments · Fixed by #284 or apache/rocketmq#6418
Closed

DLedger RPC Service thread stuck #282

RongtongJin opened this issue Mar 18, 2023 · 0 comments · Fixed by #284 or apache/rocketmq#6418
Labels
bug Something isn't working

Comments

@RongtongJin
Copy link
Contributor

RongtongJin commented Mar 18, 2023 •
edited
Loading

The following phenomena were found in our test environment

Three nodes, one node down, unable to select a leader. We found one of the DLedger nodes cannot process any rpc requests (it is candidate role), and all NettyServerPublicExecutor threads are stuck in the handlePush method

image

The number of queued tasks is increasing, but the number of completed tasks has not changed

image

The thread stuck in here, because the blocking queue is already full.

image

image

However, since it is a candidate, it will no longer peek data from the blocking queue

image
@RongtongJin RongtongJin added the bug Something isn't working label Mar 18, 2023
This was referenced Mar 18, 2023
Merged
Merged
RongtongJin added a commit that referenced this issue Mar 20, 2023
@RongtongJin
[ISSUE #282] Fix the issue that DLedger RPC Service thread stuck (#284)
234f707 
* Fix the issue that DLedger RPC Service thread stuck

* Fix the issue that DLedger RPC Service thread stuck

* Fix the issue that DLedger RPC Service thread stuck

* Fix the issue that DLedger RPC Service thread stuck
RongtongJin added a commit that referenced this issue Mar 20, 2023
@RongtongJin
[ISSUE #282] Fix the issue that DLedger RPC Service thread stuck (#283)
1388186 
* Fix the issue that DLedger RPC Service thread stuck

* Fix the check style
@RongtongJin RongtongJin mentioned this issue Mar 21, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@RongtongJin