Build crashes due to timeout or deamon not responding in time #161
Comments
|
Could you please check the logs in |
|
@ppalaga thanks for the quick reply. I check the logs and see the following. |
|
What's your operating system? Is it always reproducible? |
Funny enough, 10 minutes later -> And yes, it is reproducible. |
|
So the daemon finishes building the request, the problem is only that the client disconnects before it is finished? |
Yes, I think that's correct. The client disconnects because it has not received any message (including the keep-alive ones) for 3 seconds. That's a new behaviour for the keep-alive. |
|
Okay, indeed it is running if I add the following parameter |
|
I've been able to reproduce the problem, but in my case, it was a follow-up of #155 : when building the java doc, the daemon try to send a big message, fails with an exception, thus the client fail 3s later. |
|
@deradam the The global properties location is |
|
Yes, I'm on it. However I am fighting with the overall build now :/ |
|
Hi @gnodet, so this is the error that I am always getting when building the project. As you can see, it "breaks" when maven is calling Thanks for your support! |
|
hi together, unfortunately I have the same problem.
Problem occurs in multi-thread as well as with only one thread mode... Following system infos and the error log: Do you have any ideas? |
|
Any updates? |
|
@DoPaNik seems to be like #291 Tweaking the props should be right strategy, but the ones you mention are either not properly spelled or have weird values. As step one, I'd recommend increasing
There is a pause of 3+seconds between the following two messages in your log: Given your config, tt should be around 100ms. It would be interesting to figure what's the cause. You may want to attach jconsole or similar to the daemon process to see whether it is GC or something else. |
|
I've also hit the problem a few times under heavy cpu load. I wonder if we should slightly raise a bit the default values. |
|
I think we should try to figure out first where the bottle neck is. Is that overall CPU contention? Or is that client threads blocking each other? Daemon threads blocking each other? Not all of those potential issues would necessarily be solved by increasing mvnd.maxLostKeepAlive |
This is most probably something like a big GC pause under heavy CPU usage. I really don't think this comes from blocking threads. The threading model is not that complicated and this would happen more often I think. In addition, if there's a deadlock, raising the timeout will definitely not help. So the opposite is true: raising the timeout will remove possible problems with GC/high CPU usage why not hiding other problems, so it looks quite safe to do it anyway... |
|
Thank you for the tips. Now I have set the mvnd.properties to the following values: This looks much better at first glance. But the build breaks off after about 4 minutes with the following error: Looks like a heap size issue to me. Unfortunately, I can't connect to the jvm via jconsole at the moment. Is there a hack to start with the jconsole? I simply have tried it via jconsole UI to connect via the local connection dialog. |
Have you tried increasing the heap via mvnd.maxHeapSize ?
That's what I usually do. You should see the daemon process in the list. You may want to start the daemon eagerly via |
Yes, I also see it as the main suspect :) But it would still be nice to get a witness who sees a GC happening at that time via some monitoring tool. By increasing the defaults we just move the threshold for the client disconnect, but we do not help much those poor devs whose build is still bigger than that new threshold. Maybe we should try harder on the client side to figure out what is happening with the daemon. If the process is still up, we could perhaps prolong the grace period, but output some sort of a warning that something unusual may be going on. Also we should give a hint which particular config options should be tweaked if the user thinks the situation is normal. And finally, if somebody really sees GC as the root cause, it would be extremely interesting to see whether using shenandoah (or any other low pause gc) would help. |
Yep. I have set the heap to 8G, Which should be enough. Unfortunately still the same error. |
Of course. I try to collect data and then post them. |
|
Fun fact: With 16G started build is still running ... A normal maven build takes about 30 min. I am trying to provide data via jconsole as soon as possible |
|
I also wonder if the messages ordering on the server side isn't problematic: |
|
@ppalaga as promised, albeit a bit late, here is the output from jconsole: After I set the heap to 8 gb, the build runs without errors. But apparently mvnd had not done anything for a long time and waited in the middle of the run.... |
|
Raise the keep-alive timeout seems to no effects now. |
@ppalaga i run into a timeout issue upon today too. FWIW, in Trino, we use https://github.com/airlift/airlift/blob/7e987a2c7f55c1703b9ea383daa32c657d29b7dd/stats/src/main/java/io/airlift/stats/JmxGcMonitor.java#L77 to report GC pauses. |
It's already supposed to : https://github.com/apache/maven-mvnd/blob/master/daemon/src/main/java/org/mvndaemon/mvnd/daemon/DaemonExpiration.java
|
|
Is this still an issue? |
Hi there,
when building a rather large project (395 maven modules), mvnd runs for few minutes and then radomly "crashes" with the following error.
However if I check the running processes, I can still see mvnd building the project in the background. Is there a setting for the timeout to the deamon process?
Thanks for the nice tool and your support!
The text was updated successfully, but these errors were encountered: