Daemon crashing when building big project.

[galegofer]

I am currently trying to build a monolith with around 15 modules. With vanilla, Maven takes around 20 minutes. When I tried to build it with mvnd I am getting that somehow the daemon crashed:

13:03:16.035 I Dispatch message: KeepAlive
13:03:16.136 I Dispatch message: KeepAlive
13:03:16.236 I Dispatch message: KeepAlive
13:03:16.337 I Dispatch message: KeepAlive
13:03:16.438 I Dispatch message: KeepAlive
13:03:16.538 I Dispatch message: KeepAlive
13:03:16.639 I Dispatch message: KeepAlive
13:03:16.739 I Dispatch message: KeepAlive
----- End of the daemon log file -----
  output: C:\Users\fy95bn\.m2\mvnd\registry\0.2.0\daemon-f3678b8f-7dea-4fff-95c6-8e7915255fda.out.log
----- Last  200 lines from daemon output - C:\Users\fy95bn\.m2\mvnd\registry\0.2.0\daemon-f3678b8f-7dea-4fff-95c6-8e7915255fda.out.log -----

----- End of the daemon output -----

        at org.mvndaemon.mvnd.client.DaemonClientConnection.receive(DaemonClientConnection.java:125)
        at org.mvndaemon.mvnd.client.DefaultClient.execute(DefaultClient.java:254)
        at org.mvndaemon.mvnd.client.DefaultClient.main(DefaultClient.java:98)
Caused by: java.io.IOException: No message received within 3000ms, daemon may have crashed. You may want to check its status using mvnd --status
        at org.mvndaemon.mvnd.client.DaemonClientConnection.receive(DaemonClientConnection.java:107)
        ... 2 more

But I guess the problem is indeed when trying to ping the daemon to see if alive, because, after it crashed with the error showed above, I can see the java process is still alive (maven building)

Checking C:\Users\fy95bn.m2\mvnd\registry\0.2.0\daemon-f3678b8f-7dea-4fff-95c6-8e7915255fda.out.log:

I can see the next:

13:03:20.203 I Dispatch message: KeepAlive
13:03:20.207 E Error dispatching events
org.mvndaemon.mvnd.common.DaemonException$RecoverableMessageIOException: Could not write message KeepAlive to '/127.0.0.1:50604'.
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:116)
	at org.mvndaemon.mvnd.daemon.Server.lambda$handle$3(Server.java:468)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
	at sun.nio.ch.SocketDispatcher.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:469)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeWithNonBlockingRetry(DaemonConnection.java:275)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeBufferToChannel(DaemonConnection.java:263)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.flush(DaemonConnection.java:257)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:113)
	... 2 common frames omitted
13:03:26.011 D Expiration check running
13:03:26.014 D Storing daemon stop event: after the daemon was no longer found in the daemon registry
13:03:26.015 I Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
13:03:26.015 D Stop as soon as idle requested. The daemon is busy.
13:03:26.016 I Updating state to: StopRequested
13:03:26.016 D daemon stop has been requested. Sleeping until state changes.
13:03:36.012 D Expiration check running
13:03:46.011 D Expiration check running

[ppalaga]

It looks like the daemon was busy doing something between the last KeepAlive message the client sees at 13:03:16.739 and the last KeepAlive message the daemon attempts (and fails) to send to the client at 13:03:20.203. There was more than three seconds between them and so the client exited concluding that the daemon is dead.

The pause may have been caused by GC, by an exhaustion of CPU resources, etc. You may want to attach jcosole to the daemon JVM to see whether GC is the cause. If so, you may increase mvnd.maxLostKeepAlive in your ~/.m2/mvnd.properties. Increasing mvnd.maxHeapSize or experimenting with other garbage collectors might also help.

[famod]

I also had this at least two times while buidling Quarkus (WSL2, Ubuntu 20), the last one just a few minutes ago.
This was with a fresh daemon which is surprising because the default maxHeapSize seems to be 2G and Quarkus recommends 1.5G.
I'll try to gather some stats the next time.

[ppalaga]

@galegofer does tweaking mvnd.maxLostKeepAlive and/or mvnd.maxHeapSize help?

[galegofer]

So far it seems to be working after I upgraded to 0.3.0, even without mvnd.maxLostKeepAlive and/or mvnd.maxHeapSize.

[ppalaga]

Nice to hear that, let's close this and feel free to re-open if needed.

[famod]

Just happend to me with current master of Quarkus. Second run of mvnd -quickly.

mvnd native client 0.4.1-linux-amd64 (2336bde9192e9d0ccc3a5d2bf13abc9f295e21a7)
Terminal: org.jline.terminal.impl.PosixSysTerminal with pty org.jline.terminal.impl.jansi.linux.LinuxNativePty
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /home/moldowan/.sdkman/candidates/mvnd/0.4.1/mvn
Java version: 11.0.10, vendor: AdoptOpenJDK, runtime: /home/moldowan/.sdkman/candidates/java/11.0.10.j9-adpt
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.19.128-microsoft-standard", arch: "amd64", family: "unix"

Unfortunatly, I have no clue how to connect via jconsole because I'm using WSL2.

[ppalaga]

Just start $JAVA_HOME/bin/jconsole and use its UI to connect to the running daemon JVM

[famod]

WSL2 is headless. So I think I'll have to enable JMX and forward the ports somehow...

[gnodet]

Fwiw, I just tried mvnd -Dquickly on latest quarkus master, and it work 3 times in a row without any problem. We definitely need more data.

[hashhar]

Sorry to bump something so old but have we considered making the mvnd.maxHeapSize config default to null so that the JVM can decide the Xmx? This will bring it in line with vanilla Maven which leaves the heap size to the JVM unless an explicit -Xmx is passed?

It was a bit frustrating to discover that the daemon is crashing because it is low on memory especially since the observable failure is the lost keepalives instead of an OOM.

[gnodet]

Sorry to bump something so old but have we considered making the mvnd.maxHeapSize config default to null so that the JVM can decide the Xmx? This will bring it in line with vanilla Maven which leaves the heap size to the JVM unless an explicit -Xmx is passed?

It was a bit frustrating to discover that the daemon is crashing because it is low on memory especially since the observable failure is the lost keepalives instead of an OOM.

This is a possibility, could you open a new issue for that ?