Incremental builds

[michalszynkiewicz]

It would be great if mvnd had an option to rebuild only the modules that have been modified since they were last built successfully.

@gnodet, @ppalaga mentioned that you may be already looking into it.

If not, I might be interested in implementing it (if I have time ;) )

[gnodet]

Without having the knowledge of the actual inputs for each plugin, I don' think there's any safe way to say that a plugin (or even module) can be safely skipped. When optimizing the camel build time, I made sure that each plugin we were using could do a quick check and bypass the execution if the input had not changed.

I'm working on a Test Impact Analysis tool for maven so that the build could skip the tests that aren't affected by changes. It's still a prototype currently: https://github.com/mvndaemon/mvntia

[michalszynkiewicz]

I was thinking of simply adding some timestamp of the last build to the target directory and then skipping a rebuild of an entire module if nothing has changed in places other than target (maybe also IDE settings?)

[gnodet]

@michalszynkiewicz I'm not sure storing the timestamp in each build is a good idea
Also, I think what we're looking for has been implemented as part of https://issues.apache.org/jira/browse/MNG-5760 which is targeted at maven 4.0.
I'd rather try to port the related commits somehow to support parallel builds rather than trying to implement something completely different.

[michalszynkiewicz]

While working with Quarkus, I often modify multiple modules. It's a big project and I want a fast feedback loop so I usually handwrite a command like mvn clean install -pl <a long list of modules>.

I started thinking of writing a small tool/shell that would figure out the list of modules for me automatically.
I thought that maybe instead of doing a separate thing, it could be added to mvnd.
Do I understand correctly that you think it's not a good fit for mvnd?

[gnodet]

@michalszynkiewicz No, I definitely agree something has to be done. Until maven supports it, I agree it could be done in mvnd, but I'd rather do it the same way than maven: analyse the output, store the list of failed/skipped projects into a target/mvnd-resume.properties and use it if -r/--resume is added on the command line.

[jjlharrison]

@gnodet : Isn't there a difference between an incremental build and resuming a build? For me, when I say "incremental" I normally mean only building modules that have been changed, not just resuming from a failure.

[gnodet]

@jjlharrison yes, there's a difference, but I think the workflow described by @michalszynkiewicz is more a resume than incremental
For incremental, I have much more reservations at this point as a single timestamp would certainly not be sufficient. Let's say you change the parent pom, you have to rebuild, even if no direct input has changed. And there are much worse scenarii. Incremental build needs each plugin to actually list all its input / output so that the system can determine if there is a change.

[michalszynkiewicz]

-rf does a good job for resuming for me.
What I'm after is rebuilding modules that have changed files, not continuing from last failure.

Modifying a parent pom is indeed an example of a non-trivial change that should cause a rebuild.
Another one could be a changed BOM file that is imported by a module.
Do you see any more examples like that?

In my experience BOM or parent pom changes are much less frequent than code changes that I'd like to test, so ignoring them for now (so that one has to do a full rebuild in this case) would be an acceptable start.
And, at least these two cases, doesn't seem very complicated to handle. Maybe if a chain of parent poms/bom imports would go out of the project and back that could be not easy but I think giving up in such a case is okay.

[jjlharrison]

@gnodet: Yes, I understand that it is very complex and there can even be external dependencies that might change to affect things.

If there were a true incremental option, it would need some major caveats and the user would need to be responsible for knowing when they can use that option. I imagine it would be a lot of work to support as well.

[gnodet]

To be clear, I did experiment a bit with the takari lifecycle https://github.com/takari/takari-lifecycle which provides a real incremental replacement of the core maven. However, my experiments were not really conclusive, especially if you have an existing build which is quite complex with non basic plugins.

At the end, I optimised all the plugins that were used so that they can be really fast if nothing has changed (basically, doing incremental build at the plugin level). This is much easier than implementing a truly increment build for maven.

mvnd helps in this regard if you use the -Dbuildtime.output.log=true which provides great insight into which plugins execute slowly.

[famod]

Don't want to lure anyone away from mvnd (which is great), but for git-based incremental builds I'd like to suggest checking out https://github.com/vackosar/gitflow-incremental-builder (disclaimer: I'm maintaining that extension)

(In fact, there is some WIP to integrate it into the Quarkus build setup: quarkusio/quarkus#13243)

[gnodet]

I'm going to close this issue. I'm not opposed to an incremental build, but it should be evaluated in the light of https://github.com/takari/takari-lifecycle, the https://github.com/vackosar/gitflow-incremental-builder. Also, it would be better as a separate maven extension, which could be hosted in this organisation if the existing ones are not satisfying (provided they can't be easily enhanced).