Possible stuck thread in Jetty #3645
Comments
|
Please use Java 11.0.3 (many many networking improvements over Java 10!) and Jetty 9.4.18.v20190429 (many threading improvements over Jetty 9.4.14) and try again. |
|
Threads referenced by OP. From threaddump-1557771226203.txt @ 2019-05-13 11:13:04 From threaddump-1557771266357.txt @ 2019-05-13 11:13:52 |
|
What does your Namely the Have you run without this customization, using standard (and unconfigured, with default values) |
|
@joakime
class OTQueuedThreadPool extends InstrumentedQueuedThreadPool {
OTQueuedThreadPool(MetricRegistry metricRegistry) {
super(metricRegistry,
32,
32,
30000,
new BlockingArrayQueue<>());
}
@Override
protected void runJob(Runnable job) {
try {
job.run();
} finally {
// clean up code here
}
}Until now we have not tried running without this customization, if no viable solution is found, will do that in the future (most likely after we have tried jetty 9.4.14). Stack trace from 11.0.3 JDK, stuck thread, in this case, is default-pool-73. instance 2 |
|
Updating with latest observation: Stuck thread on instance 1 (has been in this state multiple days) Stuck thread on instance 2 (has been in this state multiple days) |
|
|
@gregw (apologies for the late response, did not notice the questions)
|
|
Got a couple of more stack traces, interestingly these are ones I have not seen in the past. Yet they occur on a thread that is running hot, stuck in a running state of which it will never come off of. Stack trace Stack trace Stack traces were obtained via YourKit. |
|
Closing this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have been running a reverse proxy based on Jetty's AsyncMiddleManServlet for years without any major issue. However, we started noticing that once in a great while a thread gets stuck in a "hot" loop. The only way to recover is to bounce the application. Whenever a server gets in this state we notice substantially (~30% more) higher CPU usage when compared to its peers, this will continue until the application is bounced.
Details:
Jetty version: 9.4.14.v20181114
JDK version: 10.0.2
Suspected stuck thread: default-pool-303663
Thread dump threaddump-1557771226203.txt @ 2019-05-13 11:13:04
Thread dump threaddump-1557771266357.txt @ 2019-05-13 11:13:52
Server state serverstate.txt
Reference email thread: https://www.eclipse.org/lists/jetty-users/msg08899.html
@sbordet please take a look, let me know if I can provide more info. I know you requested thread dumps within seconds of each other, however, this is the closest I could get it.
The text was updated successfully, but these errors were encountered: