Parallel build fails with NoSuchFileException for target/classes

[otbutz]

I'm testing mvnd on a project with a large number of modules and it seems to always fail at the same point.

Redacted output:

18:21:04  [INFO] Reactor Summary for xxxx:
18:21:04  [INFO] 
[...]
18:21:04  [INFO] project-x ............................... SUCCESS [  0.104 s]
[...]
18:21:04  [INFO] project-y ............................... FAILURE [  0.027 s]
[...]
18:17:29  [ERROR] java.nio.file.NoSuchFileException: /mnt/build/workspace/ci-branch/project-x/target/classes
18:17:29  [ERROR] project-y: java.nio.file.NoSuchFileException: /mnt/build/workspace/ci-branch/project-x/target/classes

Note that project-y has a dependency on project-x and project-x is built successfully.

mvnd version: 0.6.0

mvnd -B -T 4 compile

Simply switching back to Maven 3.8.3 solves the problem.

[gnodet]

It's a bit hard to investigate without any way to reproduce though...

[ppalaga]

When building successfully with Maven 3.8.3 do you also pass -T 4?

Does it pass with mvnd -B -T 4 compile --serial?

/mnt sounds like a mount. Is it some sort of a network (nfs, smb, etc.) mount?

[otbutz]

When building successfully with Maven 3.8.3 do you also pass -T 4?

We're building on Jenkins and i simply switched from mvn to mvnd. So all arguments are the same.

Does it pass with mvnd -B -T 4 compile --serial?

I'll try.

/mnt sounds like a mount. Is it some sort of a network (nfs, smb, etc.) mount?

Nope. Just a regular SSD formatted with ext4.

[gnodet]

@otbutz Do you have a clean goal in the command line ? It could be related to #408 but it's supposed to be fixed in 0.6.0 ...

[otbutz]

Nope. Could it be related to #484 ?

[otbutz]

Just gave it a spin with 0.8.0 and it works now :)

[gnodet]

Just gave it a spin with 0.8.0 and it works now :)

It's not an official release yet, I'll certainly have to cut it again in the coming days until the release process has been fully adapted to the move to the ASF.

[Tibor17]

@otbuz
Why you think it has to do with mvnd and/or the version 0.8.0, only because you use it?
I would like to see the stack trace using -e like this mvnd -e -B -T 4 compile.
I have always had a suspicion about Maven Project model and Resolver and the thread safety in there. Due to events are clustered together, these issues may happen for long time and disappear for long time.

[otbutz]

I might have been too early with my verdict:

08:40:40.839 I Dispatch message: KeepAlive
08:40:40.939 I Dispatch message: KeepAlive
08:40:41.039 I Dispatch message: KeepAlive
08:40:41.140 I Dispatch message: KeepAlive
08:40:41.240 I Dispatch message: KeepAlive
08:40:41.340 I Dispatch message: KeepAlive
08:40:41.441 I Dispatch message: KeepAlive
08:40:45.930 I Dispatch message: KeepAlive
08:40:46.030 I Dispatch message: KeepAlive
08:40:46.038 E Error dispatching events
org.mvndaemon.mvnd.common.DaemonException$RecoverableMessageIOException: Could not write message KeepAlive to '/127.0.0.1:41476'.
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:116)
	at org.mvndaemon.mvnd.daemon.Server.lambda$handle$3(Server.java:500)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeWithNonBlockingRetry(DaemonConnection.java:275)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeBufferToChannel(DaemonConnection.java:263)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.flush(DaemonConnection.java:257)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:113)
	... 2 common frames omitted
08:40:47.656 D Expiration check running
08:40:47.694 D Storing daemon stop event: after the daemon was no longer found in the daemon registry
08:40:47.698 I Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
08:40:47.698 D Stop as soon as idle requested. The daemon is busy.
08:40:47.698 I Updating state to: StopRequested
08:40:47.700 D daemon stop has been requested. Sleeping until state changes.
08:40:58.005 D Expiration check running

I'm going to try with 0.7.1 to rule out a regression from 0.8.0

@Tibor17 added the -e flag and will report back 😉

[gnodet]

I might have been too early with my verdict:

08:40:40.839 I Dispatch message: KeepAlive
08:40:40.939 I Dispatch message: KeepAlive
08:40:41.039 I Dispatch message: KeepAlive
08:40:41.140 I Dispatch message: KeepAlive
08:40:41.240 I Dispatch message: KeepAlive
08:40:41.340 I Dispatch message: KeepAlive
08:40:41.441 I Dispatch message: KeepAlive
08:40:45.930 I Dispatch message: KeepAlive
08:40:46.030 I Dispatch message: KeepAlive
08:40:46.038 E Error dispatching events
org.mvndaemon.mvnd.common.DaemonException$RecoverableMessageIOException: Could not write message KeepAlive to '/127.0.0.1:41476'.
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:116)
	at org.mvndaemon.mvnd.daemon.Server.lambda$handle$3(Server.java:500)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeWithNonBlockingRetry(DaemonConnection.java:275)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeBufferToChannel(DaemonConnection.java:263)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.flush(DaemonConnection.java:257)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:113)
	... 2 common frames omitted
08:40:47.656 D Expiration check running
08:40:47.694 D Storing daemon stop event: after the daemon was no longer found in the daemon registry
08:40:47.698 I Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
08:40:47.698 D Stop as soon as idle requested. The daemon is busy.
08:40:47.698 I Updating state to: StopRequested
08:40:47.700 D daemon stop has been requested. Sleeping until state changes.
08:40:58.005 D Expiration check running

I'm going to try with 0.7.1 to rule out a regression from 0.8.0

@Tibor17 added the -e flag and will report back 😉

I don't think this exception is related to the problem. The message indicates the daemon could not be found in the registry, which is usually the case when the registry is manually deleted. I also suspect the exception just being a side effect of the daemon trying to stop (because of the registry problem).

[otbutz]

0.7.1 consistently fails with the aforementioned java.nio.file.NoSuchFileException.

For 0.8.0 the build continues a lot longer but the daemon (sometimes?) just seems to die at some point:

09:04:15.324 I Dispatch message: ProjectLogMessage{projectId='xxxxxxxxxxxxxxxx', message='[INFO] [stdout] Generating code...'}
09:04:15.324 D Expiration check running
09:04:15.341 D Storing daemon stop event: after the daemon was no longer found in the daemon registry
09:04:15.329 E Error dispatching events
org.mvndaemon.mvnd.common.DaemonException$RecoverableMessageIOException: Could not write message ProjectLogMessage{projectId='xxxxxxxxxxxxxxxx', message='[INFO] [stdout] Generating code...'} to '/127.0.0.1:49960'.
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:116)
	at org.mvndaemon.mvnd.daemon.Server.lambda$handle$3(Server.java:500)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:51)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeWithNonBlockingRetry(DaemonConnection.java:275)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.writeBufferToChannel(DaemonConnection.java:263)
	at org.mvndaemon.mvnd.common.DaemonConnection$SocketOutputStream.flush(DaemonConnection.java:257)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.mvndaemon.mvnd.common.DaemonConnection.dispatch(DaemonConnection.java:113)
	... 2 common frames omitted
09:04:15.351 I Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
09:04:15.390 D Stop as soon as idle requested. The daemon is busy.
09:04:15.390 I Updating state to: StopRequested
09:04:15.402 D daemon stop has been requested. Sleeping until state changes.
09:04:25.311 D Expiration check running
09:04:33.671 I Build finished, finishing message dispatch
09:04:33.671 I Daemon back to idle
09:04:33.671 I Updating state to: Idle
09:04:36.324 D daemon is running. Sleeping until state changes.
09:04:36.324 D Expiration check running
09:04:36.331 D Storing daemon stop event: other compatible daemons were started and after being idle for 4m54s277ms and not recently used
09:04:36.332 I Daemon will be stopped at the end of the build other compatible daemons were started and after being idle for 4m54s277ms and not recently used
09:04:36.333 D Marking daemon stopped due to other compatible daemons were started and after being idle for 4m54s277ms and not recently used. The daemon is running a build: false
09:04:36.333 I Updating state to: Stopped
09:04:36.334 D daemon has stopped.
09:04:36.339 E Error running daemon loop
java.nio.channels.AsynchronousCloseException: null
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:256)
	at org.mvndaemon.mvnd.daemon.Server.accept(Server.java:223)
	at java.lang.Thread.run(Thread.java:748)

No signs of OOM killing etc

[otbutz]

Skimming through the 0.8.0 changes, i think the original issue is a duplicate of #564

As for the new error...no idea

[gnodet]

@otbutz feel free to raise a new issue for the other exception, I need to investigate if the Broken pipe exception is the cause or the consequence of the Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon message.