Make QuarkusBuild not pollute Gradle's build cache

[snazy]

Make QuarkusBuild not pollute Gradle's build cache

Improve Quarkus build in Gradle, proper up-to-date mechanisms, right config-source priorities.

Background / Motivation

Currently the QuarkusBuild task implementation adds even large build artifacts and unmodified dependency jars to Gradle's build cache. This pollutes the Gradle build cache quite a lot and therefore lets archived build caches become unnecessary huge.
On top of that, "relicts" from previous runs of QuarkusBuild using a different configuration, like a different package-type, causes multiple permutations of the cached artifacts and leads to restoring unexpected artifacts from the build cache.

This large change updates the Gradle logic to have "proper" up-to-date checks and in turn a reasonable setting of what is cached and what is not cached.

Background on how I discovered the "caching and up-to-date issue" with QuarkusBuild: Quarkus 2.16 added the @CacheableTask annotation to QuarkusBuild. This means that everything that is generated by QuarkusBuild (and "declared" as an output: fast-jar directory, uber-jar, native-runner) is cached. After having Quarkus 2.16.x in our "main" branch for about 1/2 day, nearly the whole "10 GB budget" for cache artifacts in GitHub was occupied by Gradle build cache artifacts - the build cache artifacts grew from build to build - this was not the case without the @CacheableTask annotation.

The current inputs of the QuarkusBuild task are also incomplete. Those inputs do not include for example the configured finalName, the settings on the Gradle project and the settings on the Quarkus extension. This leads to situations that the wrong Gradle cache artifacts could be restored (think: you ask for a fast-jar, but get an uber-jar).

Another issue is that repeated runs of quarkusBuild against the exact same code base and dependencies leads to different outputs, the contents of the generated jars and e.g. quarkus-application.dat differ. This causes different outputs of the QuarkusBuild task, which cause depending tasks and projects to unnecessarily run/work again.

Initial Goal

The initial goal of this effort was to make caching of fast-jar builds work nice in CI environments, I assume that's the most common package type used in CI environments for example for testing purposes. The idea here is to not cache all the dependency jars, but leverage the mechanisms available in Gradle to "just collect" the dependencies. In other words (and simplified): for fast-jar, cache everthing "in quarkus-app except lib/.

Updated Quarkus Gradle build implementation

There is no change for users running a ./gradlew quarkusBuild. QuarkusBuild is still the task that users should execute (or depend on in their builds).

To achive the goal to make especially fast-jar builds "nicely cacheable" in CI, two new tasks had to be introduced:

• QuarkusBuildDependencies uses the dependency information from the Quarkus ApplicationModel to tell Gradle to collect those. This task works for fast-jar and legacy-jar and is a "no op" for other package types. This task is never cacheable, but has up-to-date checks.
• QuarkusBuildCacheableAppParts performs a Quarkus application build and collects the cacheable parts of that build. This task works for fast-jar and legacy-jar and is a "no op" for other package types. This task is cacheable.
• QuarkusBuild, for fast-jar and legacy-jar, assembles the outputs of the above two tasks to produce the expected output. Performs a Quarkus build for other package types. See below on when this task is cacheable.

The build/quarkus-build/ directory is used to properly separate the outputs of the above three tasks:

• build/quarkus-build/gen/ receives the full output of a Quarkus build
• build/quarkus-build/app/ receives the cacheable parts from build/quarkus-build/gen/
• build/quarkus-build/dep/ receives the dependency jars

CI vs non-CI

The output of QuarkusBuild is, by default, cacheable in non-CI environments and not cacheable in CI environments. The behavior can be explicitly overridden using the new property cacheLargeArtifacts property. As outlined above, caching huge artifacts is not beneficial in CI, either due to space limitations (GitHub's 10GB limit for example) or just the cost of network traffic/duration. On a developer's machine however, the cost of storing/retrieving even bigger artifacts is rather neglectible and lower than rebuilding a Quarkus application.

Configuration

Before this change, the "priority" of configuration settings was (in most cases...) effectively: System properties, environment variables, application.properties, configurations in Gradle build scripts & Gradle project properties. This order is unintuitive and was changed to: system properties, environment variables, Gradle project properties, Quarkus build configuration, application.properties. The previous code had several places in which Quarkus related configuration settings were retrieved with sometimes different priorities of the "config sources".

The whole "machinery" to get the configuration has been rewritten and encapsulated in new classes EffectiveConfig (effective for a Quarkus build) and BaseConfig (available during Gradle task configuration phase). EffectiveConfig is not really different from BaseConfig, but contains the "special" settings from e.g. ImagePush` task.

The new implementation also uses SmallRye Config and especially the existing Quarkus mechanisms to pull information out of a PackageConfig object produced from a configuration. It turned out to be easier to reason about PackageConfig than "raw property values".

Support for application.yaml/yml has also been added.

Workers

All calls into the "inner workings" of a Quarkus build and Quarkus code generation have been moved to separate worker processes. The code in this change to support this is pretty dead simple.

The primary reason to move that work into isolated worker processes is that all the QuarkusBootstrap and derived pieces know nothing about Gradle, so there is no "property like" mechanism to override settings from application.properties/yaml/yml with those from e.g. the QuarkusPluginExtension. The only option would have been to modify the system properties of the Gradle build process - but that's a no-go, especially considering other tasks running in parallel (think: two QuarkusBuild of different projects running at the same time). It would have been relatively easy to serialize all QuarkusBuild actions across a Gradle build, but why - and it would prevent using "beefy machines" to run many QuarkusBuilds in parallel. Another option would have been to implement a rather complex (and likely very racy) mechanism to track modifications to system properties.

As a conclusion, it wasn't just very simple to leverage the process isolation using the Gradle worker API, but it's also not bad. Gradle does reuse already spawned and compatible worker instances.

The "trick" implemented to "prioritize" configs from Gradle project properties and Quarkus extension settings is to pass the whole config as system properties to the worker.

Logging / cache-disabled messages

A bunch of hopefully useful logging has been implemented and added.

With info-level loogging, the tasks emit at least some basic information about what they do and at least the package type being used.

Effective config

To be able to investigate which configuration settings were effectively used, there's another new task called quarkusShowEffectiveConfig that shows all the quarkus.* properties and some more information, including the loaded application.(properties|yaml|yml). This task is intended to debug build issues. The task can optionally save the effecitve config properties as a properties file in the build/ directory, if run with the command line option ./gradlew quarkusShowEffectiveConfig --save-config-properties.

Test changes

None of the existing tests has changed, except the BuildCOnfigurationTest had to be adopted to reflect the updated order of config sources.

Relates to: #30852

[github-actions]

🙈 The PR is closed and the preview is expired.

[snazy]

@snazy could you disable those tests on windows?

Sure, done + rebased

[snazy]

Hum - so that's not nice :(

Finally found some hint in the logs:

> Task :quarkusGenerateCode
Generating Quarkus code for org.test:test-codestart:1.0.0-codestart
  launch mode:                  NORMAL
  base name:                    test-codestart-1.0.0-codestart
  generated source directory:   D:\a\quarkus\quarkus\integration-tests\devtools\target\quarkus-codestart-build-test\project-gradle-kotlin-dsl-java-d770bd4b-e0d6-4c8b-a697-ebe48232953a\build\classes\java\quarkus-generated-sources
  build directory:              D:\a\quarkus\quarkus\integration-tests\devtools\target\quarkus-codestart-build-test\project-gradle-kotlin-dsl-java-d770bd4b-e0d6-4c8b-a697-ebe48232953a\build
'"java.exe"' is not recognized as an internal or external command,

operable program or batch file.
--q-grpc_out: protoc-gen-q-grpc: Plugin failed with status code 1.


FAILURE: Build failed with an exception.
> Task :quarkusGenerateCode FAILED

I've reverted the test-ignores for Windows, added the --stracktrace argument to some more Gradle invocations and (temporarily) added unconditional logging of the error stacktrace.

The error above comes from some code invoked by io.quarkus.gradle.tasks.worker.CodeGenWorker. Not sure whether the double and single quotes around java.exe are some hint though.

[snazy]

For posterity: some gRPC/codegen integration tests fail(ed) only on Windows, so likely an OS specific issue, but not 100% clear where and why exactly. On a "fresh" Windows machine with just a Java + Git installed, and Java w/o all the environments like JAVA_HOME, PATH and w/o the registry keys configured, the tests failing in Windows-CI failed on that machine as well. Some results from experiments:

• JAVA_HOME is mandatory for both Maven + Gradle to find the Java executable (not really a surprise)
• Having JAVA_HOME\bin in PATH is neither necessary for Maven nor for Gradle (also not really a surprise). But it is necessary for at least the grpc code generation, likely the codegen binaries.
• Populating both JAVA_HOME + PATH to the worker processes seems to fix it - as implemented in GradleTask.configureProcessWorkerSpec().

[aloubyansky]

@snazy it seems to be passing now. Could you please squash the commits? Hopefully, that'll be it for this one!

[quarkus-bot]

Failing Jobs - Building ee0f74f

Status	Name	Step	Failures	Logs	Raw logs
✔️	Gradle Tests - JDK 11
⌛	Gradle Tests - JDK 11 Windows	Build	⚠️ Check →	Logs	Raw logs
✖	JVM Tests - JDK 11	Build	Failures	Logs	Raw logs
✔️	JVM Tests - JDK 17
⌛	JVM Tests - JDK 19	Build	⚠️ Check →	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ JVM Tests - JDK 11 #

- Failing: extensions/resteasy-reactive/quarkus-resteasy-reactive-jackson/deployment 
! Skipped: extensions/keycloak-admin-client-reactive/deployment extensions/oidc-client-reactive-filter/deployment extensions/oidc-token-propagation-reactive/deployment and 45 more

📦 extensions/resteasy-reactive/quarkus-resteasy-reactive-jackson/deployment

✖ io.quarkus.resteasy.reactive.jackson.deployment.test.ExceptionInWriterTest.test line 27 - More details - Source on GitHub

java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)

[gsmet]

Thanks @snazy , hopefully, this won't break the Internet :).

[snazy]

Hehe - I already download that thing today ;)

[galderz]

This doesn't seem to be working as expected, or maybe it worked at the time but no longer does. See #39347

The more worrying thing is that changes like this should have been reflected in the integration tests, but looking at NativeIntegrationTestIT, it's still using testNative.