Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build time performance regression and bigger native binaries when migrating from 3.5 to 3.6 or 3.7 #38683

Closed
zakkak opened this issue Feb 8, 2024 · 43 comments · Fixed by #38886
Milestone

Comments

@zakkak
Copy link
Contributor

zakkak commented Feb 8, 2024

Describe the bug

This was originally reported in https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/performance.20decrease.20with.20graalvm.2023.2E1.2E2.20-quarkus.203.2E5.20to.203.2E6 by @vsevel

hello, I am trying to upgrade one of our app that is running today with 3.2
I am seeing big perf decrease on the native build, specifically between quarkus 3.5 and 3.6
up to 3.5 (mandrel 23.0.3), I get native builds with around 10Gb of Peak RSS and augmentation time around 450 secs
starting with 3.6 (mandrel 23.1.2), I get native builds with around 15Gb of Peak RSS and augmentation time around 900 secs

this seems related to the mandrel version change.

With Quarkus 3.5 and Mandrel 23.0.3

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.0.3.0 JDK 17.0.10+7
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-JpspI quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dlogging.initial-configurator.min-level=500 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.quarkus.caffeine.graalvm.recordStats=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature,io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json --allow-incomplete-classpath -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback -H:+ReportExceptionStackTraces -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/reflect-config\.json --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register a feature using the option --features=oracle.nativeimage.NativeImageFeature
[1/8] Initializing...                                                                                   (40.5s @ 0.44GB)
 Java version: 17.0.10+7, vendor version: Mandrel-23.0.3.0-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 6 user-specific feature(s)
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
 - io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - oracle.nativeimage.NativeImageFeature
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
15:30:29,939 WARN  [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis...  [**********]                                                             (125.1s @ 3.84GB)
  26,311 (77.92%) of 33,766 types reachable
  49,848 (70.67%) of 70,540 fields reachable
 170,497 (67.72%) of 251,765 methods reachable
   9,987 types, 3,332 fields, and 51,258 methods registered for reflection
      71 types,    93 fields, and    56 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (17.6s @ 4.14GB)
[4/8] Parsing methods...      [****]                                                                    (15.0s @ 3.62GB)
[5/8] Inlining methods...     [****]                                                                     (5.3s @ 4.28GB)
[6/8] Compiling methods...    [*********]                                                               (80.9s @ 7.62GB)
[7/8] Layouting methods...    [****]                                                                    (13.4s @ 5.20GB)
[8/8] Creating image...       [*****]                                                                   (21.9s @ 6.63GB)
  60.22MB (50.47%) for code area:   123,816 compilation units
  58.71MB (49.20%) for image heap:  653,443 objects and 136 resources
 410.26kB ( 0.34%) for other data
 119.33MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  13.52MB java.base                                           13.26MB byte[] for code metadata
   6.00MB svm.jar (Native Image)                               6.89MB java.lang.Class
   3.84MB xx-5.1.1-SNAPSHOT-runner.jar                  6.31MB byte[] for java.lang.String
   3.33MB io.fabric8.kubernetes-model-core-6.8.1.jar           5.90MB java.lang.String
   2.18MB org.freemarker.freemarker-2.3.32.jar                 4.99MB byte[] for general heap data
   1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar         4.67MB byte[] for reflection metadata
   1.32MB modified-io.vertx.vertx-core-4.4.6.jar               2.25MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   1.28MB java.net.http                                        2.21MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.24MB io.fabric8.openshift-model-6.8.1.jar                 1.22MB java.lang.String[]
 627.07kB java.naming                                        953.16kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
  23.95MB for 265 more packages                                9.87MB for 5682 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                       27.2s (8.2% of total time) in 92 GCs | Peak RSS: 10.41GB | CPU load: 4.18
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/xx-5.1.1-SNAPSHOT-runner (executable)
 /project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 5m 23s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 388452ms

With Quarkus 3.6 and Mandrel 23.1.2

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.1.2.0 JDK 21.0.2+13-LTS
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-ORizE quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dlogging.initial-configurator.min-level=500 -J-Dio.quarkus.caffeine.graalvm.recordStats=true -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature,io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap --allow-incomplete-classpath -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
Warning: Using a deprecated option --allow-incomplete-classpath from command line. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where t...
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register a feature using the option --features=oracle.nativeimage.NativeImageFeature
[1/8] Initializing...                                                                                   (52.4s @ 0.42GB)
 Java version: 21.0.2+13-LTS, vendor version: Mandrel-23.1.2.0-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 6 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
 - io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - oracle.nativeimage.NativeImageFeature
------------------------------------------------------------------------------------------------------------------------
 4 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:-UseServiceLoaderFeature' (origin(s): command line)
 - '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///project/lib/io.micrometer.micrometer-core-1.11.5.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 17.78GB of memory (70.9% of 25.06GB system memory, set via '-Xmx20g')
 - 6 thread(s) (100.0% of 6 available processor(s), determined at start)
15:47:00,469 WARN  [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis...  [*****]                                                                  (468.2s @ 9.26GB)
   44,742 reachable types   (93.8% of   47,723 total)
   91,492 reachable fields  (81.3% of  112,513 total)
  403,198 reachable methods (84.6% of  476,837 total)
   28,302 types, 57,105 fields, and 238,786 methods registered for reflection
       69 types,    91 fields, and    56 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (26.1s @ 9.78GB)
[4/8] Parsing methods...      [*****]                                                                  (27.2s @ 10.15GB)
[5/8] Inlining methods...     [****]                                                                    (11.7s @ 9.41GB)
[6/8] Compiling methods...    [************]                                                           (141.5s @ 5.30GB)
[7/8] Layouting methods...    [******]                                                                  (39.4s @ 7.28GB)
[8/8] Creating image...       [********]                                                                (66.7s @ 8.28GB)
 154.67MB (48.39%) for code area:   352,852 compilation units
 164.56MB (51.48%) for image heap:1,238,079 objects and 200 resources
 418.96kB ( 0.13%) for other data
 319.63MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  18.46MB io.fabric8.openshift-model-operator-6.9.2.jar       48.89MB byte[] for code metadata
  16.82MB svm.jar (Native Image)                              28.81MB byte[] for java.lang.String
  14.59MB io.fabric8.openshift-model-6.9.2.jar                22.70MB java.lang.Class
  13.77MB java.base                                           19.13MB byte[] for reflection metadata
   9.21MB io.fabric8.kubernetes-model-core-6.9.2.jar          11.04MB java.lang.String
   4.83MB io.fabric8.openshift-model-monitoring-6.9.2.jar     10.87MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.80MB io.fabric8.openshift-model-hive-6.9.2.jar            3.75MB com.oracle.svm.core.hub.DynamicHubCompanion
   4.20MB xx-5.1.1-SNAPSHOT-runner.jar                  2.65MB java.lang.Object[]
   2.98MB i.f.k.9.2.jar                                        2.31MB java.lang.String[]
   2.78MB io.fabric8.openshift-model-operatorhub-6.9.2.jar     1.77MB byte[] for general heap data
  59.56MB for 267 more packages                               12.65MB for 5792 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                      129.8s (15.4% of total time) in 249 GCs | Peak RSS: 15.80GB | CPU load: 4.53
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/xx-5.1.1-SNAPSHOT-runner (executable)
 /project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 13m 57s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 933775ms

Expected behavior

Build times and binary size are expected to be comparable between 3.5, 3.6 and 3.7 given no other changes happened in the application code.

Actual behavior

The user observes significant increases in build execution times, memory usage, and binary size.

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

Mandrel or GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

@zakkak zakkak added kind/bug Something isn't working area/native-image labels Feb 8, 2024
Copy link

quarkus-bot bot commented Feb 8, 2024

You added a link to a Zulip discussion, please make sure the description of the issue is comprehensive and doesn't require accessing Zulip

This message is automatically generated by a bot.

Copy link

quarkus-bot bot commented Feb 8, 2024

/cc @Karm (mandrel), @galderz (mandrel), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

@zakkak
Copy link
Contributor Author

zakkak commented Feb 8, 2024

Looking at the provided build output one can already see some things::

  1. The time spent in the analysis is way longer in 23.1 than in 23.0 and that's reflected in the reachable types. What's interesting is that not only have the reachable types increased as a % of the total types, but the total reachable types also increase (same for fields and methods).
  2. There is a big increase in the types registered for reflection.

Note that these differences do not match the ones we observed in https://quarkus.io/blog/mandrel-23-1-image-size-increase/

It looks like in Q 3.6 we are building double the code...

If I understand correctly the way this was tested is by building the exact same code with Quarkus 3.5 and then with Quarkus 3.6, which means that you use both a different set of dependencies and a different builder image. Could you please run another build with Quarkus 3.6 but this time with the jdk-17 builder image? i.e. passing -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17

This will allow us to see if the main difference is due to the different Mandrel builder image or due to some Quarkus change.

It looks like what you are seeing is also related to #37142

You might want to give #37278 a try:

This PR allows for the exclusion of openshift-model-hive, openshift-model-miscellaneous, openshift-model-operator in native mode, which will already downsize the resulting binary considerably.

Indeed running with Quarkus 3.7.1 and Mandrel 23.0.3 gives similar results, indicating that this is not caused by the Mandrel update, but by some dependency change (e.g. "in Quarkus 3.5.3, we are using io.fabric8 6.8.1, and 6.9.2 in 3.6"):

$ mvn clean package -o -DskipTests -Dnative -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on MANDREL 23.0.3.0 JDK 17.0.10+7
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --name build-native-KGynW quay.io/quarkus/ub
i-quarkus-mandrel-builder-image:jdk-17 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.quarkus.caffein
e.graalvm.recordStats=true -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetec
tion.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.hibernate.validator.runtime.DisableLoggingFeature,io.quarkus.r
untime.graal.DisableLoggingFeature,io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/
java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=xx-5.1.1-SNAPSHOT-runner-build-output-stats.json --allow-incomplete-classpath -H
:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback -H:+ReportExceptionStackTraces -J-Xmx20g -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:-UseServiceLoaderFeature -H:+StackTrace -J-
-add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config com\.oracle\.database\.jdbc /META-INF/native-image/native-image\.properties --exclude-config com\.oracle\.database\.jdbc /META-INF/native-imag
e/reflect-config\.json --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-
handler/generated/handlers/reflect-config\.json xx-5.1.1-SNAPSHOT-runner -jar xx-5.1.1-SNAPSHOT-runner.jar
Warning: Using a deprecated option --allow-incomplete-classpath from command line. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
========================================================================================================================
GraalVM Native Image: Generating 'xx-5.1.1-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register
a feature using the option --features=oracle.nativeimage.NativeImageFeature
...
[1/8] Initializing...                                                                                   (54.9s @ 0.46GB)
 Java version: 17.0.10+7, vendor version: Mandrel-23.0.3.0-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 6 user-specific feature(s)
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
 - io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - oracle.nativeimage.NativeImageFeature
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
The bundle named: oracle.net.mesg.Message, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
17:17:51,730 WARN  [fre.dom] No XPath support is available. If you need it, add Apache Xalan or Jaxen as dependency.
[2/8] Performing analysis...  [*****]                                                                  (865.3s @ 8.90GB)
  44,303 (94.03%) of 47,115 types reachable
  91,352 (81.78%) of 111,702 fields reachable
 402,379 (84.68%) of 475,176 methods reachable
  28,167 types, 57,447 fields, and 240,611 methods registered for reflection
      71 types,    93 fields, and    56 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (82.7s @ 8.96GB)
GC warning: 47.0s spent in 5 GCs during the last stage, taking up 56.49% of the time.
            Please ensure more than 14.26GB of memory is available for Native Image
            to reduce GC overhead and improve image build time.
[4/8] Parsing methods...      [******]                                                                 (44.4s @ 10.07GB)
[5/8] Inlining methods...     [****]                                                                    (13.3s @ 8.63GB)
[6/8] Compiling methods...    [***************]                                                        (240.5s @ 5.06GB)
[7/8] Layouting methods...    [******]                                                                  (44.2s @ 5.89GB)
[8/8] Creating image...       [*******]                                                                 (55.8s @ 6.98GB)
 152.88MB (50.89%) for code area:   353,694 compilation units
 147.16MB (48.98%) for image heap:1,233,195 objects and 138 resources
 410.15kB ( 0.13%) for other data
 300.43MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  18.32MB io.fabric8.openshift-model-operator-6.10.0.jar      31.86MB byte[] for code metadata
  16.92MB svm.jar (Native Image)                              22.61MB java.lang.Class
  14.28MB io.fabric8.openshift-model-6.10.0.jar               22.18MB byte[] for java.lang.String
  13.53MB java.base                                           19.25MB byte[] for reflection metadata
   9.11MB io.fabric8.kubernetes-model-core-6.10.0.jar         10.97MB java.lang.String
   4.70MB io.fabric8.openshift-model-monitoring-6.10.0.jar    10.95MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.68MB io.fabric8.openshift-model-hive-6.10.0.jar           8.33MB byte[] for general heap data
   4.66MB xx-5.1.1-SNAPSHOT-runner.jar                  3.72MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.90MB i.f.k.10.0.jar                                       2.58MB java.lang.Object[]
 /project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 23m 26s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm -v /e/git/java-xx/target/xx-5.1.1-SNAPSHOT-native-image-source-jar:/project:z --entrypoint /bin/bash quay.io/quarkus/ubi-q
uarkus-mandrel-builder-image:jdk-17 -c objcopy --strip-debug xx-5.1.1-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 1527193ms
[INFO]
[INFO] --- build-helper-maven-plugin:3.0.0:attach-artifact (attach-artifacts) @ xx ---
[INFO] Skip attaching artifacts
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:21 min
[INFO] Finished at: 2024-02-07T18:38:06+01:00
[INFO] ------------------------------------------------------------------------

@vsevel also reported that:

I was able to get some savings with the referenced exclusions (Quarkus 3.7.1 ):

75.9s (11.4% of total time) in 148 GCs | Peak RSS: 12.05GB | CPU load: 4.50
...
Finished generating 'x-5.1.1-SNAPSHOT-runner' in 10m 56s.

better than without the exclusion, still not as good compared to 3.5 :

 27.2s (8.2% of total time) in 92 GCs | Peak RSS: 10.41GB | CPU load: 4.18
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/xx-5.1.1-SNAPSHOT-runner (executable)
 /project/xx-5.1.1-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'xx-5.1.1-SNAPSHOT-runner' in 5m 23s.

I am going to try to see if I can add more exclusions.
still I am very much surprised about the impact additional libraries can have on the build.

@gsmet
Copy link
Member

gsmet commented Feb 8, 2024

FWIW, I know that at some point a Kubernetes Client update caused some problems as the amount of code was massive.

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

I am continuing trying to get rid of as many models as possible. it is a pain, because some missing models appear at native runtime only. so it is a lengthy process.

@zakkak
Copy link
Contributor Author

zakkak commented Feb 8, 2024

FWIW, I know that at some point a Kubernetes Client update caused some problems as the amount of code was massive.

Correct, that was #37142

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

with the latest exclusions in 3.7.1 I get on our CI:

49.4s (14.7% of total time) in 171 GCs | Peak RSS: 11.30GB | CPU load: 6.04
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 35s.

on a test branch for 3.5 I get:

23.4s (10.0% of total time) in 115 GCs | Peak RSS: 8.32GB | CPU load: 5.77
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 53s.

our baseline on 3.2 (master) :

30.2s (13.2% of total time) in 167 GCs | Peak RSS: 5.83GB | CPU load: 5.91
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 48s.

the increase of memory is scary. our build machines are going to have a hard time coping with processes 11Gb of RAM.

here is the exclusions I was able to do:

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-openshift-client</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-hive</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-miscellaneous</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-operator</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-clusterautoscaling</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-operatorhub</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-machine</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-whereabouts</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-storageversionmigrator</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-tuned</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-console</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-machineconfig</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>openshift-model-installer</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

and

        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>certmanager-client</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-resource</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-rbac</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-admissionregistration</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-autoscaling</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-coordination</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-discovery</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-events</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-flowcontrol</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-metrics</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-policy</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-scheduling</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-storageclass</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.fabric8</groupId>
                    <artifactId>kubernetes-model-node</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

even with all that cleanup, this shows a signifiant increase in memory and build time.
what else can I do?

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

3.5:

2024.02.07 12:58:38  [2/8] Performing analysis...  [**********]                                                              (89.6s @ 3.25GB)
2024.02.07 12:58:38    25,787 (77.52%) of 33,265 types reachable
2024.02.07 12:58:38    49,039 (70.80%) of 69,265 fields reachable
2024.02.07 12:58:38   168,338 (67.65%) of 248,837 methods reachable
2024.02.07 12:58:38     9,796 types, 2,542 fields, and 51,144 methods registered for reflection
2024.02.07 12:58:38        71 types,    93 fields, and    56 methods registered for JNI access
2024.02.07 12:58:38         4 native libraries: dl, pthread, rt, z
2024.02.07 12:58:48  [3/8] Building universe...                                                                              (12.8s @ 3.43GB)
2024.02.07 12:58:58  [4/8] Parsing methods...      [***]                                                                     (10.7s @ 3.57GB)
2024.02.07 12:59:06  [5/8] Inlining methods...     [****]                                                                     (6.8s @ 2.94GB)
2024.02.07 13:00:14  [6/8] Compiling methods...    [*******]                                                                 (57.7s @ 5.28GB)
2024.02.07 13:00:19  [7/8] Layouting methods...    [****]                                                                    (14.7s @ 3.91GB)
2024.02.07 13:00:41  [8/8] Creating image...       [****]                                                                    (20.2s @ 5.33GB)
2024.02.07 13:00:41    59.47MB (50.26%) for code area:   122,614 compilation units
2024.02.07 13:00:41    58.45MB (49.40%) for image heap:  644,721 objects and 136 resources
2024.02.07 13:00:41   408.55kB ( 0.34%) for other data
2024.02.07 13:00:41   118.31MB in total
2024.02.07 13:00:41  ------------------------------------------------------------------------------------------------------------------------
2024.02.07 13:00:41  Top 10 origins of code area:                                Top 10 object types in image heap:
2024.02.07 13:00:41    13.52MB java.base                                           13.09MB byte[] for code metadata
2024.02.07 13:00:41     5.99MB svm.jar (Native Image)                               6.76MB java.lang.Class
2024.02.07 13:00:41     3.84MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar                  6.22MB byte[] for java.lang.String
2024.02.07 13:00:41     3.33MB io.fabric8.kubernetes-model-core-6.8.1.jar           5.81MB java.lang.String
2024.02.07 13:00:41     2.18MB org.freemarker.freemarker-2.3.32.jar                 4.92MB byte[] for general heap data
2024.02.07 13:00:41     1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar         4.61MB byte[] for reflection metadata
2024.02.07 13:00:41     1.32MB modified-io.vertx.vertx-core-4.4.6.jar               2.25MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.07 13:00:41     1.28MB java.net.http                                        2.16MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.07 13:00:41     1.23MB io.fabric8.openshift-model-6.8.1.jar                 1.21MB java.lang.String[]
2024.02.07 13:00:41   627.05kB java.naming                                        934.53kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
2024.02.07 13:00:41    23.24MB for 264 more packages                                9.79MB for 5560 more object types

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

3.7

2024.02.08 17:24:43  [2/8] Performing analysis...  [*****]                                                                  (138.7s @ 5.92GB)
2024.02.08 17:24:43     32,715 reachable types   (91.6% of   35,730 total)
2024.02.08 17:24:43     62,938 reachable fields  (75.3% of   83,560 total)
2024.02.08 17:24:43    270,083 reachable methods (78.6% of  343,567 total)
2024.02.08 17:24:43     16,500 types, 29,030 fields, and 133,756 methods registered for reflection
2024.02.08 17:24:43         69 types,    91 fields, and    56 methods registered for JNI access
2024.02.08 17:24:43          4 native libraries: dl, pthread, rt, z
2024.02.08 17:24:55  [3/8] Building universe...                                                                              (22.2s @ 6.12GB)
2024.02.08 17:25:07  [4/8] Parsing methods...      [***]                                                                     (11.4s @ 4.77GB)
2024.02.08 17:25:14  [5/8] Inlining methods...     [****]                                                                     (6.3s @ 7.43GB)
2024.02.08 17:26:36  [6/8] Compiling methods...    [*********]                                                               (80.0s @ 4.02GB)
2024.02.08 17:27:03  [7/8] Layouting methods...    [*****]                                                                   (27.5s @ 7.24GB)
2024.02.08 17:27:35  [8/8] Creating image...       [*****]                                                                   (31.9s @ 4.55GB)
2024.02.08 17:27:35   107.24MB (50.12%) for code area:   224,347 compilation units
2024.02.08 17:27:35   106.34MB (49.69%) for image heap:  901,740 objects and 200 resources
2024.02.08 17:27:35   417.65kB ( 0.19%) for other data
2024.02.08 17:27:35   213.99MB in total
2024.02.08 17:27:35  ------------------------------------------------------------------------------------------------------------------------
2024.02.08 17:27:35  Top 10 origins of code area:                                Top 10 object types in image heap:
2024.02.08 17:27:35    14.59MB io.fabric8.openshift-model-6.10.0.jar               33.65MB byte[] for code metadata
2024.02.08 17:27:35    13.77MB java.base                                           17.27MB byte[] for java.lang.String
2024.02.08 17:27:35    10.05MB svm.jar (Native Image)                              11.33MB java.lang.Class
2024.02.08 17:27:35     9.33MB io.fabric8.kubernetes-model-core-6.10.0.jar         10.98MB byte[] for reflection metadata
2024.02.08 17:27:35     4.83MB io.fabric8.openshift-model-monitoring-6.10.0.jar     8.13MB java.lang.String
2024.02.08 17:27:35     4.75MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar                  6.12MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.08 17:27:35     2.98MB i.f.k.10.0.jar                                       2.75MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.08 17:27:35     2.87MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar    1.70MB java.lang.Object[]
2024.02.08 17:27:35     2.24MB org.freemarker.freemarker-2.3.32.jar                 1.70MB java.lang.String[]
2024.02.08 17:27:35     2.21MB io.fabric8.certmanager-model-v1alpha2-6.10.0.jar     1.49MB byte[] for general heap data
2024.02.08 17:27:35    37.92MB for 245 more packages                               11.22MB for 5683 more object types

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

3.2

2024.02.07 22:55:47  [2/8] Performing analysis...  [*********]                                                               (90.9s @ 2.90GB)
2024.02.07 22:55:47    26,276 (73.80%) of 35,603 types reachable
2024.02.07 22:55:47    49,003 (70.23%) of 69,779 fields reachable
2024.02.07 22:55:47   167,491 (64.73%) of 258,751 methods reachable
2024.02.07 22:55:47    10,020 types, 2,067 fields, and 48,594 methods registered for reflection
2024.02.07 22:55:47        71 types,    93 fields, and    56 methods registered for JNI access
2024.02.07 22:55:47         4 native libraries: dl, pthread, rt, z
2024.02.07 22:55:58  [3/8] Building universe...                                                                              (12.5s @ 3.39GB)
2024.02.07 22:56:08  [4/8] Parsing methods...      [***]                                                                     (11.0s @ 2.04GB)
2024.02.07 22:56:13  [5/8] Inlining methods...     [****]                                                                     (4.1s @ 1.91GB)
2024.02.07 22:57:09  [6/8] Compiling methods...    [*******]                                                                 (55.3s @ 3.86GB)
2024.02.07 22:57:25  [7/8] Layouting methods...    [****]                                                                    (14.6s @ 2.77GB)
2024.02.07 22:57:43  [8/8] Creating image...       [****]                                                                    (18.8s @ 2.92GB)
2024.02.07 22:57:43    59.40MB (50.50%) for code area:   120,579 compilation units
2024.02.07 22:57:43    57.81MB (49.16%) for image heap:  645,267 objects and 134 resources
2024.02.07 22:57:43   412.78kB ( 0.34%) for other data
2024.02.07 22:57:43   117.61MB in total
2024.02.07 22:57:43  ------------------------------------------------------------------------------------------------------------------------
2024.02.07 22:57:43  Top 10 origins of code area:                                Top 10 object types in image heap:
2024.02.07 22:57:43    13.61MB java.base                                           13.10MB byte[] for code metadata
2024.02.07 22:57:43     6.02MB svm.jar (Native Image)                               6.94MB java.lang.Class
2024.02.07 22:57:43     3.76MB ocpdeploy-5.1.1-SNAPSHOT-runner.jar                  6.26MB byte[] for java.lang.String
2024.02.07 22:57:43     2.97MB io.fabric8.kubernetes-model-core-6.7.2.jar           5.83MB java.lang.String
2024.02.07 22:57:43     2.18MB org.freemarker.freemarker-2.3.32.jar                 4.49MB byte[] for general heap data
2024.02.07 22:57:43     1.93MB c.f.jackson.core.jackson-databind-2.15.2.jar         4.43MB byte[] for reflection metadata
2024.02.07 22:57:43     1.30MB modified-io.vertx.vertx-core-4.4.4.jar               2.21MB com.oracle.svm.core.hub.DynamicHubCompanion
2024.02.07 22:57:43     1.28MB java.net.http                                        2.13MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
2024.02.07 22:57:43     1.14MB io.fabric8.openshift-model-6.7.2.jar                 1.21MB java.lang.String[]
2024.02.07 22:57:43   629.42kB java.naming                                        944.41kB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
2024.02.07 22:57:43    23.58MB for 270 more packages                                9.64MB for 5594 more object types

@vsevel
Copy link
Contributor

vsevel commented Feb 8, 2024

tests on 3.6 and 3.7 were run with a large xmx (20Gb). I reran the tests with the original xmx=7g that we have on master, and it changes the numbers I gave earlier:

3.7

55.1s (15.7% of total time) in 283 GCs | Peak RSS: 7.51GB | CPU load: 6.00
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 50s.

3.6

57.8s (16.6% of total time) in 308 GCs | Peak RSS: 7.21GB | CPU load: 5.38
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 5m 46s.

baseline 3.2

30.2s (13.2% of total time) in 167 GCs | Peak RSS: 5.83GB | CPU load: 5.91
Finished generating 'ocpdeploy-5.1.1-SNAPSHOT-runner' in 3m 48s.

not perfect, but memory looks far better.

@manusa
Copy link
Contributor

manusa commented Feb 9, 2024

even with all that cleanup, this shows a significant increase in memory and build time.
what else can I do?

I'm not 100% sure, but I'd say that with those exclusions, the client models shouldn't be accountable for the increase in size.

Take the kubernetes-model-core as an example (which is fairly stable).

The original jar size vs the reported origins of code area:

  • 3.2 / 6.7.2 4907714:2.97MB
  • 3.5 / 6.8.1 3665233:3.33MB
  • 3.7 / 6.10.0 3631645:9.33MB

Even when the original jar file is reducing its size (we did split it up at some point), the resulting size accounted for these types keeps increasing.

@zakkak
Copy link
Contributor Author

zakkak commented Feb 9, 2024

@vsevel looking at the 3.5 vs 3.7 build outputs (I am assuming you are using Mandrel 23.0 in 3.5 and Mandrel 23.1 in 3.7) I see that both the code area and the heap size increased.

For the code area increase (by ~50MB) the key contributors appear to be:
1. io.fabric8.openshift-model-6.10.0.jar (is ~13MB larger than it used to be in 6.8.1)
2. svm.jar (is ~4MB larger than it used to in Mandrel 23.0)
3. io.fabric8.kubernetes-model-core-6.10.0.jar (is ~6MB larger than it used to be in 6.8.1)
4. io.fabric8.openshift-model-monitoring-6.10.0.jar (brings at least 3MB extra)
5. io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar (brings at least 2MB extra)
6. ...

For the heap area increase (by ~50MB) the key contributors appear to be:
1. 20MB of extra metadata (part of this is explained in https://quarkus.io/blog/mandrel-23-1-image-size-increase/)
2. 11MB of extra String values
3. 6MB of extra metadata used for reflection (This is caused by registering more things for reflection)
4. 5MB of extra Classes
5. ...

All the above indicate that more code has become reachable and is leading in the build process to be more resource hungry and to take longer. It also results in a larger binary in the end. I think that at this point there is not much you can do on the application side, but we also can't to much without more info.

It would be great if you could share a reproducer with us, but if that's not possible it would also help if you could run a 3.5 and a 3.7 build and a quick run afterwards with the following parameters and share the output files with us.

parameters

-Dquarkus.native.additional-build-args=-R:+DumpHeapAndExit,-H:+DashboardAll,-H:+DashboardJson,-H:DashboardDump=path/to/dashboard.json
  1. -Dquarkus.native.enable-reports

interesting files

  1. target/your-app-native-image-sources/your-app-build-output-stats.json (created at build time)
  2. your-app.hprof (created after running the native executable of your app)
  3. target/your-app-native-image-sources/reports (this is a directory please include all files)
  4. dashboard.json.dump (created at build time)

@zakkak
Copy link
Contributor Author

zakkak commented Feb 9, 2024

@vsevel I forgot to mention that in order to reduce the moving parts to the minimum it would be better to build only using Mandrel 23.0 for the time being.

@adinn
Copy link

adinn commented Feb 14, 2024

One way to identify more about what is increasing build time RSS would be to generate heap dumps during build and see which object types/how many object instances are being held on to. This might indicate which component of GraalVM Native is blowing up.

Heap dumps might also indicate whether the RSS increase is down to a GraalVM/Mandrel change or -- as looks more likely -- a Quarkus change. If you interpolate between old and new configs by generating heap dumps with the Quarkus 3.7.1 and Mandrel 23.0 then that might further clarify which of the two components is responsible (could still be both).

@metacosm
Copy link
Contributor

I have run some quick tests on a sample operator using QOSDK 6.3.x which targets Quarkus 3.2.10 and QOSDK main with Quarkus 3.7.3:

3.2.10:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on native-image 21.0.2 2024-01-16
GraalVM Runtime Environment GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)
Substrate VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13, serial gc)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http,https -H:-UseServiceLoaderFeature -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner.jar
Warning: The option '-H:+AllowFoldMethods' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:-UseServiceLoaderFeature' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
WARNING: Unknown module: org.graalvm.sdk specified to --add-exports
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                   (12.3s @ 0.35GB)
 Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
 Graal compiler: optimization level: 2, target machine: armv8-a
 C compiler: cc (apple, arm64, 15.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
 4 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:-UseServiceLoaderFeature' (origin(s): command line)
 - '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.1.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 26.49GB of memory (41.4% of 64.00GB system memory, determined at start)
 - 10 thread(s) (100.0% of 10 available processor(s), determined at start)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
        at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
        at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
        at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
        at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
        at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
        at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
java.lang.IllegalAccessError: class io.quarkus.runner.Feature (in unnamed module @0x6ab50d1c) cannot access class org.graalvm.nativeimage.impl.RuntimeClassInitializationSupport (in module org.graalvm.nativeimage) because module org.graalvm.nativeimage does not export org.graalvm.nativeimage.impl to unnamed module @0x6ab50d1c
        at io.quarkus.runner.Feature.runtimeReinitializedClasses(Unknown Source)
        at io.quarkus.runner.Feature.beforeAnalysis(Unknown Source)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:773)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
[2/8] Performing analysis...  [16:04:11,601 WARN  [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*********]                                                               (25.8s @ 2.03GB)
   17,190 reachable types   (69.7% of   24,678 total)
   33,054 reachable fields  (70.9% of   46,590 total)
  122,784 reachable methods (63.3% of  193,960 total)
    7,173 types,   871 fields, and 40,747 methods registered for reflection
       61 types,    61 fields, and    55 methods registered for JNI access
        5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe...                                                                               (3.9s @ 2.32GB)
[4/8] Parsing methods...      [**]                                                                       (2.7s @ 2.95GB)
[5/8] Inlining methods...     [***]                                                                      (1.2s @ 2.55GB)
[6/8] Compiling methods...    [****]                                                                    (16.7s @ 3.16GB)
[7/8] Layouting methods...    [***]                                                                      (5.4s @ 2.30GB)
[8/8] Creating image...       [***]                                                                      (8.0s @ 3.78GB)
  40.02MB (46.83%) for code area:    89,175 compilation units
  44.47MB (52.04%) for image heap:  447,396 objects and 56 resources
 992.39kB ( 1.13%) for other data
  85.46MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  11.78MB java.base                                           13.50MB byte[] for code metadata
   4.76MB svm.jar (Native Image)                               6.85MB byte[] for java.lang.String
   2.87MB io.fabric8.kubernetes-model-core-6.7.2.jar           4.38MB java.lang.Class
   1.80MB c.f.jackson.core.jackson-databind-2.15.2.jar         4.06MB java.lang.String
   1.37MB q.3.6-SNAPSHOT-runner.jar                            3.52MB byte[] for reflection metadata
   1.24MB io.fabric8.openshift-model-6.7.2.jar                 1.78MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   1.10MB modified-io.vertx.vertx-core-4.4.6.jar               1.44MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.08MB org.bouncycastle.bcprov-jdk18on-1.74.jar           860.73kB byte[] for general heap data
 562.08kB io.fabric8.openshift-model-operator-6.7.2.jar      846.55kB java.lang.String[]
 561.55kB com.fasterxml.jackson.core.jackson-core-2.15.2.jar 754.49kB java.lang.Object[]
  12.49MB for 134 more packages                                6.53MB for 3534 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 INIT: Adopt '--strict-image-heap' to prepare for the next GraalVM release.
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                        6.5s (8.3% of total time) in 103 GCs | Peak RSS: 6.32GB | CPU load: 5.44
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner (executable)
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.3.6-SNAPSHOT-runner' in 1m 17s.

3.7.3:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.0.2+13-jvmci-23.1-b30
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (8.0s @ 0.43GB)
 Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
 Graal compiler: optimization level: 2, target machine: armv8-a
 C compiler: cc (apple, arm64, 15.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
 3 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:-UseServiceLoaderFeature' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 26.49GB of memory (41.4% of 64.00GB system memory, determined at start)
 - 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis...  [16:21:48,585 WARN  [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*****]                                                                  (191.1s @ 8.51GB)
   34,517 reachable types   (94.8% of   36,416 total)
   73,068 reachable fields  (84.1% of   86,892 total)
  339,522 reachable methods (86.5% of  392,604 total)
   24,339 types, 51,922 fields, and 216,678 methods registered for reflection
       61 types,    61 fields, and    55 methods registered for JNI access
        5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe...                                                                              (7.2s @ 11.65GB)
[4/8] Parsing methods...      [***]                                                                      (6.5s @ 7.95GB)
[5/8] Inlining methods...     [***]                                                                      (2.8s @ 9.26GB)
[6/8] Compiling methods...    [******]                                                                  (39.5s @ 7.39GB)
[7/8] Layouting methods...    [****]                                                                    (14.9s @ 5.84GB)
[8/8] Creating image...       [*****]                                                                   (23.8s @ 5.06GB)
 120.43MB (46.23%) for code area:   303,936 compilation units
 137.73MB (52.88%) for image heap:  999,594 objects and 57 resources
   2.33MB ( 0.89%) for other data
 260.49MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  17.00MB io.fabric8.openshift-model-operator-6.10.0.jar      40.09MB byte[] for code metadata
  14.62MB svm.jar (Native Image)                              24.68MB byte[] for java.lang.String
  13.48MB io.fabric8.openshift-model-6.10.0.jar               19.74MB java.lang.Class
  12.10MB java.base                                           17.11MB byte[] for reflection metadata
   8.61MB io.fabric8.kubernetes-model-core-6.10.0.jar          9.87MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.47MB io.fabric8.openshift-model-monitoring-6.10.0.jar     8.92MB java.lang.String
   4.44MB io.fabric8.openshift-model-hive-6.10.0.jar           2.90MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.78MB i.f.k.10.0.jar                                       2.38MB java.lang.Object[]
   2.65MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar    1.85MB java.lang.String[]
   2.59MB io.fabric8.openshift-model-operatorhub-6.10.0.jar    1.48MB byte[] for general heap data
  36.07MB for 137 more packages                                8.72MB for 3626 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                       22.9s (7.8% of total time) in 95 GCs | Peak RSS: 16.90GB | CPU load: 6.61
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json
 ========================================================================================================================
 Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 4m 55s.

So while it's not completely comparing apples to apples, things do appear significantly worse in 3.7.3, with the openshift model dominating the code areas, as opposed to java.base in 3.2. Types registered for reflection are significantly worse in 3.7.3 as well.

@metacosm
Copy link
Contributor

With 3.5.0:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.2
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating CSV for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/manifests/quarkus-operator-sdk-samples-exposedapp.clusterserviceversion.yaml
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating annotations for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/metadata/annotations.yaml
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating bundle for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/bundle.Dockerfile
[INFO] [io.quarkiverse.operatorsdk.bundle.deployment.BundleProcessor] Generating Custom Resource Definition for 'quarkus-operator-sdk-samples-exposedapp' controller -> /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/bundle/quarkus-operator-sdk-samples-exposedapp/manifests/exposedapps.halkyon.io-v1.crd.yml
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (8.0s @ 0.29GB)
 Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
 Graal compiler: optimization level: 2, target machine: armv8-a
 C compiler: cc (apple, arm64, 15.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
 4 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:-UseServiceLoaderFeature' (origin(s): command line)
 - '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.1.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 13.88GB of memory (21.7% of 64.00GB system memory, determined at start)
 - 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis...  [17:00:23,192 WARN  [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
**********]                                                              (25.0s @ 3.12GB)
   17,235 reachable types   (74.2% of   23,226 total)
   33,982 reachable fields  (71.3% of   47,683 total)
  126,631 reachable methods (67.5% of  187,468 total)
    7,120 types, 1,206 fields, and 43,751 methods registered for reflection
       61 types,    61 fields, and    55 methods registered for JNI access
        5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe...                                                                               (4.9s @ 4.50GB)
[4/8] Parsing methods...      [**]                                                                       (2.6s @ 3.96GB)
[5/8] Inlining methods...     [***]                                                                      (1.2s @ 3.60GB)
[6/8] Compiling methods...    [****]                                                                    (16.9s @ 2.99GB)
[7/8] Layouting methods...    [***]                                                                      (5.6s @ 4.28GB)
[8/8] Creating image...       [***]                                                                      (9.0s @ 2.15GB)
  41.52MB (46.91%) for code area:    93,175 compilation units
  46.00MB (51.97%) for image heap:  458,097 objects and 57 resources
1014.56kB ( 1.12%) for other data
  88.51MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  12.07MB java.base                                           13.96MB byte[] for code metadata
   4.92MB svm.jar (Native Image)                               6.96MB byte[] for java.lang.String
   3.17MB io.fabric8.kubernetes-model-core-6.8.1.jar           4.55MB java.lang.Class
   1.80MB c.f.jackson.core.jackson-databind-2.15.2.jar         4.16MB java.lang.String
   1.39MB q.6.4-SNAPSHOT-runner.jar                            3.75MB byte[] for reflection metadata
   1.34MB io.fabric8.openshift-model-6.8.1.jar                 1.93MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   1.15MB org.bouncycastle.bcprov-jdk18on-1.76.jar             1.45MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.11MB modified-io.vertx.vertx-core-4.4.5.jar               1.15MB byte[] for general heap data
 630.71kB io.fabric8.openshift-model-operator-6.8.1.jar      865.65kB java.lang.String[]
 561.52kB com.fasterxml.jackson.core.jackson-core-2.15.2.jar 765.25kB java.lang.Object[]
  12.98MB for 137 more packages                                6.51MB for 3606 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                        5.9s (7.9% of total time) in 62 GCs | Peak RSS: 7.52GB | CPU load: 5.32
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 1m 14s.

@metacosm
Copy link
Contributor

With 3.6.0:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.0.2+13-jvmci-23.1-b30
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Users/claprun/.sdkman/candidates/java/21.0.2-graalce/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dlogging.initial-configurator.min-level=500 -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Duser.language=en -J-Duser.country=FR -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner -jar quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner.jar
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (9.1s @ 0.33GB)
 Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
 Graal compiler: optimization level: 2, target machine: armv8-a
 C compiler: cc (apple, arm64, 15.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
------------------------------------------------------------------------------------------------------------------------
 4 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:-UseServiceLoaderFeature' (origin(s): command line)
 - '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.micrometer.micrometer-core-1.11.5.jar', 'META-INF/native-image/io.netty/netty-transport/native-image.properties' in 'file:///Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-transport-4.1.100.Final.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 11.40GB of memory (17.8% of 64.00GB system memory, determined at start)
 - 10 thread(s) (100.0% of 10 available processor(s), determined at start)
[2/8] Performing analysis...  [17:08:25,416 WARN  [io.net.res.dns.DnsServerAddressStreamProviders] Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'
*****]                                                                  (156.8s @ 6.09GB)
   34,253 reachable types   (94.7% of   36,152 total)
   72,364 reachable fields  (84.1% of   86,072 total)
  336,541 reachable methods (86.5% of  389,146 total)
   24,179 types, 51,594 fields, and 215,026 methods registered for reflection
       61 types,    61 fields, and    55 methods registered for JNI access
        5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] Building universe...                                                                              (10.9s @ 6.70GB)
[4/8] Parsing methods...      [**]                                                                       (4.5s @ 8.24GB)
[5/8] Inlining methods...     [***]                                                                      (2.2s @ 7.43GB)
[6/8] Compiling methods...    [******]                                                                  (37.5s @ 4.05GB)
[7/8] Layouting methods...    [****]                                                                    (13.6s @ 6.02GB)
[8/8] Creating image...       [*****]                                                                   (21.7s @ 7.56GB)
 119.17MB (46.16%) for code area:   301,358 compilation units
 136.66MB (52.94%) for image heap:  992,513 objects and 57 resources
   2.32MB ( 0.90%) for other data
 258.14MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  17.00MB io.fabric8.openshift-model-operator-6.9.2.jar       39.68MB byte[] for code metadata
  14.52MB svm.jar (Native Image)                              24.52MB byte[] for java.lang.String
  13.48MB io.fabric8.openshift-model-6.9.2.jar                19.63MB java.lang.Class
  12.08MB java.base                                           16.97MB byte[] for reflection metadata
   8.49MB io.fabric8.kubernetes-model-core-6.9.2.jar           9.80MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.47MB io.fabric8.openshift-model-monitoring-6.9.2.jar      8.85MB java.lang.String
   4.44MB io.fabric8.openshift-model-hive-6.9.2.jar            2.87MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.78MB i.f.k.9.2.jar                                        2.37MB java.lang.Object[]
   2.59MB io.fabric8.openshift-model-operatorhub-6.9.2.jar     1.84MB java.lang.String[]
   2.32MB io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar    1.47MB byte[] for general heap data
  35.39MB for 137 more packages                                8.65MB for 3624 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                       24.7s (9.6% of total time) in 126 GCs | Peak RSS: 12.17GB | CPU load: 6.19
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner (executable)
 /Volumes/CaseSensitiveDev/quarkus-operator-sdk/samples/exposedapp/target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner' in 4m 17s.

@galderz
Copy link
Member

galderz commented Feb 15, 2024

Thanks for the info Chris. Indeed the operator presents the same symptoms as seen in Vincent's application. We'll look into. To replicate:

git clone https://github.com/quarkiverse/quarkus-operator-sdk
cd quarkus-operator-sdk
mvn install -DskipTests
cd samples/exposedapp
mvn package -DskipTests -Dnative -Dnative.surefire.skip

p.s. @metacosm Why doesn't the sample operator app honor -DskipTests? I had to run with mvn debug to discover that -skipTests can only be overriden with -Dnative.surefire.skip?

@metacosm
Copy link
Contributor

p.s. @metacosm Why doesn't the sample operator app honor -DskipTests? I had to run with mvn debug to discover that -skipTests can only be overriden with -Dnative.surefire.skip?

That looks like a bug, will look into it

@metacosm
Copy link
Contributor

metacosm commented Feb 15, 2024

More information, I ran native builds with reports and here's the difference of packages between 3.5 and 3.7:

--- reports-3.5/used_packages_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_095352.txt	2024-02-15 09:53:52
+++ reports-3.7/used_packages_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_102428.txt	2024-02-15 10:24:28
@@ -113,6 +113,8 @@
 com.sun.management
 com.sun.management.internal
 com.sun.naming.internal
+com.sun.org.apache.xerces.internal.jaxp.datatype
+com.sun.org.apache.xerces.internal.util
 com.sun.proxy.jdk.proxy1
 com.sun.proxy.jdk.proxy2
 com.sun.proxy.jdk.proxy3
@@ -149,10 +151,11 @@
 io.fabric8.kubernetes.api.model.events.v1
 io.fabric8.kubernetes.api.model.events.v1beta1
 io.fabric8.kubernetes.api.model.extensions
-io.fabric8.kubernetes.api.model.flowcontrol.v1alpha1
+io.fabric8.kubernetes.api.model.flowcontrol.v1
 io.fabric8.kubernetes.api.model.flowcontrol.v1beta1
 io.fabric8.kubernetes.api.model.flowcontrol.v1beta2
 io.fabric8.kubernetes.api.model.flowcontrol.v1beta3
+io.fabric8.kubernetes.api.model.gatewayapi.v1
 io.fabric8.kubernetes.api.model.gatewayapi.v1alpha2
 io.fabric8.kubernetes.api.model.gatewayapi.v1beta1
 io.fabric8.kubernetes.api.model.internal.apiserver.v1alpha1
@@ -171,6 +174,7 @@
 io.fabric8.kubernetes.api.model.scheduling.v1
 io.fabric8.kubernetes.api.model.scheduling.v1beta1
 io.fabric8.kubernetes.api.model.storage
+io.fabric8.kubernetes.api.model.storage.v1alpha1
 io.fabric8.kubernetes.api.model.storage.v1beta1
 io.fabric8.kubernetes.api.model.version
 io.fabric8.kubernetes.client
@@ -207,11 +211,14 @@
 io.fabric8.openshift.api.model.console.v1
 io.fabric8.openshift.api.model.console.v1alpha1
 io.fabric8.openshift.api.model.hive.agent.v1
+io.fabric8.openshift.api.model.hive.alibabacloud.v1
 io.fabric8.openshift.api.model.hive.aws.v1
 io.fabric8.openshift.api.model.hive.azure.v1
 io.fabric8.openshift.api.model.hive.baremetal.v1
 io.fabric8.openshift.api.model.hive.gcp.v1
 io.fabric8.openshift.api.model.hive.ibmcloud.v1
+io.fabric8.openshift.api.model.hive.metricsconfig.v1
+io.fabric8.openshift.api.model.hive.none.v1
 io.fabric8.openshift.api.model.hive.openstack.v1
 io.fabric8.openshift.api.model.hive.ovirt.v1
 io.fabric8.openshift.api.model.hive.v1
@@ -239,6 +246,7 @@
 io.fabric8.openshift.api.model.miscellaneous.cncf.cni.v1
 io.fabric8.openshift.api.model.miscellaneous.imageregistry.operator.v1
 io.fabric8.openshift.api.model.miscellaneous.metal3.v1alpha1
+io.fabric8.openshift.api.model.miscellaneous.metal3.v1beta1
 io.fabric8.openshift.api.model.miscellaneous.network.operator.v1
 io.fabric8.openshift.api.model.monitoring.v1
 io.fabric8.openshift.api.model.monitoring.v1alpha1
@@ -325,6 +333,7 @@
 io.netty.util.internal
 io.netty.util.internal.logging
 io.netty.util.internal.shaded.org.jctools.queues
+io.netty.util.internal.shaded.org.jctools.queues.atomic
 io.netty.util.internal.shaded.org.jctools.util
 io.quarkiverse.operatorsdk.common
 io.quarkiverse.operatorsdk.runtime
@@ -368,6 +377,7 @@
 io.quarkus.runtime.util
 io.quarkus.smallrye.context.runtime
 io.quarkus.smallrye.health.runtime
+io.quarkus.vertx
 io.quarkus.vertx.core.runtime
 io.quarkus.vertx.core.runtime.config
 io.quarkus.vertx.core.runtime.context
@@ -387,10 +397,10 @@
 io.smallrye.common.classloader
 io.smallrye.common.constraint
 io.smallrye.common.expression
-io.smallrye.common.function
 io.smallrye.common.ref
 io.smallrye.common.vertx
 io.smallrye.config
+io.smallrye.config._private
 io.smallrye.config.common
 io.smallrye.config.common.utils
 io.smallrye.config.inject
@@ -437,6 +447,7 @@
 io.vertx.core.impl.verticle
 io.vertx.core.json
 io.vertx.core.json.impl
+io.vertx.core.json.jackson
 io.vertx.core.logging
 io.vertx.core.metrics
 io.vertx.core.net
@@ -454,6 +465,7 @@
 io.vertx.core.spi.transport
 io.vertx.core.streams
 io.vertx.core.streams.impl
+io.vertx.core.tracing
 io.vertx.ext.web
 io.vertx.ext.web.client
 io.vertx.ext.web.handler
@@ -523,6 +535,8 @@
 javax.security.auth.callback
 javax.security.auth.x500
 javax.tools
+javax.xml.datatype
+javax.xml.namespace
 jdk.internal.access
 jdk.internal.event
 jdk.internal.icu.impl
@@ -550,6 +564,7 @@
 jdk.vm.ci.aarch64
 jdk.vm.ci.code
 jdk.vm.ci.meta
+jdk.xml.internal
 org.HdrHistogram
 org.bouncycastle.asn1
 org.bouncycastle.asn1.cms
@@ -564,6 +579,7 @@
 org.bouncycastle.crypto.constraints
 org.bouncycastle.crypto.digests
 org.bouncycastle.crypto.engines
+org.bouncycastle.crypto.modes
 org.bouncycastle.crypto.params
 org.bouncycastle.jcajce.provider.symmetric.util
 org.bouncycastle.jcajce.util
@@ -661,6 +677,7 @@
 org.snakeyaml.engine.v2.tokens
 org.wildfly.common
 org.wildfly.common._private
+org.wildfly.common.cpu
 org.wildfly.common.lock
 org.wildfly.common.math
 org.wildfly.common.net
@@ -688,6 +705,7 @@
 sun.management
 sun.misc
 sun.net
+sun.net.dns
 sun.net.ext
 sun.net.spi
 sun.net.util

Looks like for some reason, we're now getting io.quarkus.vertx and XML support, which wasn't previously the case. That said, that doesn't explain why the OpenShift model jars are now so much larger.

@metacosm
Copy link
Contributor

metacosm commented Feb 15, 2024

Upon further investigation it looks like Graal now looks at Builder classes when it previously wasn't the case so, for example, for the AdmissionReview class, the following methods are now considered by Graal, when they previously were not:

com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_0777984050acf10da1493a6170fecab4d6751fcb():AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_0fd11b86103ae5deac94f17cd7ebc9ab0f80db23(AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_1b146ff3d728cebdeee6ce3b7a16914bc1fa167a(AdmissionReviewFluent, AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_60c83db85be99307ac1cfa5e02331b73895992f2(AdmissionReviewFluent):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_7d86b4847991d0ea7bc237011e756b6bbf79ade5(AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_8407036dc50873034984857788b3f548e6463c6d():AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_b76e0fbe8c4fc902f7f35602d471fc03f7a1b049(AdmissionReviewFluent, AdmissionReview):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewBuilder_constructor_d1c32c1d0681c1b2f19acb7ac28dac86d0b61674(AdmissionReviewFluent):AdmissionReviewBuilder
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent$ResponseNested_constructor_259ca1fdf585b79b107e52e1c9e93f64803f3dd1(AdmissionReviewFluent, AdmissionResponse):AdmissionReviewFluent$ResponseNested
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent$ResponseNested_constructor_9ede732b5de68aeabe21301372523139b271aeb0(AdmissionReviewFluent, AdmissionResponse):AdmissionReviewFluent$ResponseNested
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_8e18a7129d7cfce282cc44631a106586b9f50160():AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_a3286f0b1cb894d801b61242fb3ee01a918d9ac7():AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_b36cf7bb0e633398005c903e018b111dd2f934d5(AdmissionReview):AdmissionReviewFluent
com.oracle.svm.core.code.FactoryMethodHolder.AdmissionReviewFluent_constructor_c44ba2db71b9766138f88aac8a2d5f191200375d(AdmissionReview):AdmissionReviewFluent

So with around as many methods added for each type, that certainly explains the size difference. The question, now, is why are the builders considered when they previously weren't.

@galderz
Copy link
Member

galderz commented Feb 15, 2024

Thanks @metacosm for your comments. We have replicated the issue now on our side but the amount of data to get through is considerable. The only thing I've been able to observe so far is that the blow out is already present in the Quarkus operator, which makes things easier on our side. We'll reply when we have a good understanding of the causes.

@metacosm
Copy link
Contributor

I'm looking into the reason why the builders are now considered. I think that we'll be in good shape if we can figure this out and address this particular issue first.

@manusa
Copy link
Contributor

manusa commented Feb 15, 2024

For some extra insights.
I'm working on a demo project to showcase how to work on OpenShift with just the vanilla kubernetes-client-extension: https://github.com/marcnuri-demo/quarkus-kubernetes-client-models/blob/c4123613e94646018d2d4e6ade36267dde056d8a/

Compiling a native image in 3.7.3, produces the following output:

 159.55MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  14.53MB io.fabric8.openshift-model-6.10.0.jar               25.02MB byte[] for code metadata
  12.94MB java.base                                           13.04MB byte[] for java.lang.String
   9.29MB io.fabric8.kubernetes-model-core-6.10.0.jar          8.99MB byte[] for reflection metadata
   8.82MB svm.jar (Native Image)                               8.19MB java.lang.Class
   2.97MB i.f.k.10.0.jar                                       5.88MB java.lang.String
   2.85MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar    5.15MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   2.48MB io.fabric8.kubernetes-model-flowcontrol-6.10.0.jar   1.84MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.97MB c.f.jackson.core.jackson-databind-2.16.1.jar         1.36MB java.lang.Object[]
   1.76MB io.fabric8.kubernetes-model-autoscaling-6.10.0.jar   1.24MB byte[] for general heap data
   1.75MB i.f.kubernetes-model-apiextensions-6.10.0.jar        1.21MB java.lang.String[]
  19.24MB for 117 more packages                                7.27MB for 3416 more object types

While the same project in 3.5.3:

  76.36MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  12.90MB java.base                                           11.76MB byte[] for code metadata
   4.06MB svm.jar (Native Image)                               5.93MB byte[] for java.lang.String
   3.43MB io.fabric8.kubernetes-model-core-6.8.1.jar           3.95MB java.lang.Class
   1.93MB c.f.jackson.core.jackson-databind-2.15.3.jar         3.65MB java.lang.String
   1.26MB io.fabric8.openshift-model-6.8.1.jar                 2.58MB byte[] for reflection metadata
   1.17MB modified-io.vertx.vertx-core-4.4.6.jar               1.27MB com.oracle.svm.core.hub.DynamicHubCompanion
 823.45kB q.0.0-SNAPSHOT-runner.jar                            1.21MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
 582.35kB com.fasterxml.jackson.core.jackson-core-2.15.3.jar   1.12MB byte[] for general heap data
 575.69kB org.yaml.snakeyaml-2.1.jar                         754.34kB java.lang.String[]
 510.70kB org.snakeyaml.snakeyaml-engine-2.6.jar             672.46kB java.lang.Object[]
   9.09MB for 117 more packages                                6.23MB for 3391 more object types

So the increase in size is also extremely visible here. Since this project is much smaller, maybe it's easier to use it to analyze why now classes such as builders are now being considered.

@metacosm
Copy link
Contributor

I've pinpointed the culprit: we added an edit method to all model objects in fabric8io/kubernetes-client@a3efaa3.
Prior to that commit, a native build would report:

more target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/reports/used_methods_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_164459.txt |
 grep Builder | wc -l
    6398

With that commit, we get:

 more target/quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-native-image-source-jar/reports/used_methods_quarkus-operator-sdk-samples-exposedapp-6.6.4-SNAPSHOT-runner_20240215_170845.txt |
 grep Builder | wc -l
   44403

Looking at the changelog and the symptoms, it made sense that this change was the culprit but I wanted to have the numbers to back that claim, first… 😄
Now, the question is how to fix it.

@vsevel
Copy link
Contributor

vsevel commented Feb 15, 2024

I was working on providing the resources asked by @zakkak in #38683 (comment)
I would not be able to make it public (only through a support case).
but it feels that you made a lot of progress in the analysis.
do you still need something from me?

@metacosm
Copy link
Contributor

@vsevel you probably don't need to anymore… though it might be a good idea to keep the work you did around just in case we need it later on! 😉

@galderz
Copy link
Member

galderz commented Feb 15, 2024

To follow up on ideas for a solution here:

An even better solution would be to avoid using reflection in the Kubernetes client.

Hibernate was in a not so dissimilar position 4 years ago, when pushing the boundaries of number of entities created was causing build times to increase in a big way (see Zulip chat). The refactoring fabric8io/kubernetes-client@a3efaa3 alone is not an issue. Combining that with the fact that builders get registered for reflection is what causes the blowout in build times...etc. You can test the effects of reflection registration with this small project I created at the time: reflection-bigbang

@metacosm
Copy link
Contributor

To be honest, I'm not sure why we register the builders for reflection. We do need to register the other classes of the model because we need to be able to serialize / deserialize them. Of course, it would be better to only register what is actually needed but that's just not feasible because when we have no way of knowing which classes are actually used when a given model package is imported, so all of them need to be registered.

@gsmet
Copy link
Member

gsmet commented Feb 15, 2024

It would be surprising if we needed to register the builders for reflection.

I would start by not doing that and see how it goes.

@metacosm
Copy link
Contributor

Yeah, that's what I'm planning on exploring tomorrow.

@Sanne
Copy link
Member

Sanne commented Feb 15, 2024

We do need to register the other classes of the model because we need to be able to serialize / deserialize them.

Just curious, is that actual Java serialization? It would really be nice to find a way to avoid registering such large models, especially as there's not even a guaranteed that they will ever be used.

In other extensions we worked hard to avoid reflection - it's not a strict rule as we're pragmatic about it (having some won't harm), but we do try to find alternative solutions when the number is significant.

@manusa
Copy link
Contributor

manusa commented Feb 16, 2024

Great job, @metacosm 🙌 🙌 🙌

I can't really think of a case where builders would be accessed reflectively. Maybe @shawkins has more ideas about this.

especially as there's not even a guaranteed that they will ever be used

The problem is the way in which the Fabric8 client is designed, especially when dealing with the API server or resources that users might store locally.

For example, a user might try to deserialize a YAML file which contains a variety of resources by invoking kubernetesClient.load(yamlListInputStream). The deserializer tries to figure out the target types and deserialize it into object instances of these types.
Similar situations happen when interacting with the kube API server and retrieving objects, especially deeply nested types.
This doesn't play well with GraalVM and the way it traverses the code to discover the needed types at runtime.

If removing the builders from the list of reflectively accessed classes fixes the issue, I think we might be able to get out of the woods for now.

@metacosm
Copy link
Contributor

Hmm, looks like this might not be as trivial as removing the builders from reflection since it seems they were added to solve #27469. Will dig deeper.

@metacosm
Copy link
Contributor

I have opened a PR in the fabric8 client to reduce reflective access but the issue is that the Builders are part of the type of resources that are being registered for reflection (they implement an interface that includes the associated builder as a generic parameter) so I suspect that they're being pulled transitively.

@metacosm
Copy link
Contributor

I have also opened #38823 to remove the explicit registration of builders but this doesn't help. I'm going to be on PTO all of next week, though…

@zakkak
Copy link
Contributor Author

zakkak commented Feb 19, 2024

I suspect that they're being pulled transitively.

That's right. In Quarkus main JacksonProcessor is registering 7896 types because they belong in the type hierarchy of io.fabric8.kubernetes.api.model.ValidationSchema which is annotated with @JsonDeserialize

Note that these registrations are unconditional and register for reflection all the type methods and fields as well, which leads to the explosion we observe.

To confirm this I disabled such registrations with:

diff --git a/extensions/kubernetes-client/deployment/src/main/java/io/quarkus/kubernetes/client/deployment/KubernetesClientProcessor.java b/extensions/kubernetes-client/deployment/src/main/java/io/quarkus/kubernetes/client/deployment/KubernetesClientProcessor.java
index 6c3ddb8b118..67411a4d7fe 100644
--- a/extensions/kubernetes-client/deployment/src/main/java/io/quarkus/kubernetes/client/deployment/KubernetesClientProcessor.java
+++ b/extensions/kubernetes-client/deployment/src/main/java/io/quarkus/kubernetes/client/deployment/KubernetesClientProcessor.java
@@ -26,6 +26,7 @@
 import io.fabric8.kubernetes.api.model.KubeSchema;
 import io.fabric8.kubernetes.api.model.KubernetesResource;
 import io.fabric8.kubernetes.api.model.KubernetesResourceList;
+import io.fabric8.kubernetes.api.model.ValidationSchema;
 import io.fabric8.kubernetes.client.Config;
 import io.fabric8.kubernetes.client.CustomResource;
 import io.fabric8.kubernetes.client.DefaultKubernetesClient;
@@ -67,6 +68,7 @@ public class KubernetesClientProcessor {
     private static final DotName RESOURCE_EVENT_HANDLER = DotName
             .createSimple(io.fabric8.kubernetes.client.informers.ResourceEventHandler.class.getName());
     private static final DotName KUBERNETES_RESOURCE = DotName.createSimple(KubernetesResource.class.getName());
+    private static final DotName VALIDATION_SCHEMA = DotName.createSimple(ValidationSchema.class.getName());
     private static final DotName KUBERNETES_RESOURCE_LIST = DotName
             .createSimple(KubernetesResourceList.class.getName());
     private static final DotName KUBE_SCHEMA = DotName.createSimple(KubeSchema.class.getName());
@@ -189,6 +191,7 @@ public void process(ApplicationIndexBuildItem applicationIndex, CombinedIndexBui
         ignoredJsonDeserializationClasses.produce(new IgnoreJsonDeserializeClassBuildItem(KUBE_SCHEMA));
         ignoredJsonDeserializationClasses.produce(new IgnoreJsonDeserializeClassBuildItem(KUBERNETES_RESOURCE_LIST));
         ignoredJsonDeserializationClasses.produce(new IgnoreJsonDeserializeClassBuildItem(KUBERNETES_RESOURCE));
+        ignoredJsonDeserializationClasses.produce(new IgnoreJsonDeserializeClassBuildItem(VALIDATION_SCHEMA));
 
         final String[] deserializerClasses = fullIndex
                 .getAllKnownSubclasses(DotName.createSimple("com.fasterxml.jackson.databind.JsonDeserializer"))

that saved ~50MB on its own with the reproducer in #38683 (comment)

going one step further and applying #38823 it saved another ~10MB. Which still results in a binary ~17MB bigger than in 3.5.

@manusa
Copy link
Contributor

manusa commented Feb 19, 2024

That's right. In Quarkus main JacksonProcessor is registering 7896 types because they belong in the type hierarchy of io.fabric8.kubernetes.api.model.ValidationSchema which is annotated with @JsonDeserialize

This is a great find. For me it was difficult to understand why now just by adding an edit method was pulling the builder types while before it wasn't even if the types aren't used.

This clearly explains why we're pulling everything in, regardless of that being needed at runtime or not. So #38823 shouldn't be applicable and might actually bring some corner-case scenarios.

AFAIK I think we can safely remove the ValidationSchema types, I'm not really aware of anyone actually using those (especially from a reflective perspective).

@maxandersen
Copy link
Member

@zakkak any chance you havea similar output so we can compare the sizes on each of the model jars to the other runs?

@zakkak
Copy link
Contributor Author

zakkak commented Feb 20, 2024

@zakkak any chance you havea similar output so we can compare the sizes on each of the model jars to the other runs?

Quarkus 3.5.3

[8/8] Creating image...       [**]                                                                       (4.0s @ 7.18GB)
  36.26MB (50.59%) for code area:    73,146 compilation units
  35.02MB (48.86%) for image heap:  398,582 objects and 10 resources
 402.88kB ( 0.55%) for other data
  71.68MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  12.83MB java.base                                            8.02MB byte[] for code metadata
   4.02MB svm.jar (Native Image)                               3.90MB java.lang.Class
   3.28MB io.fabric8.kubernetes-model-core-6.8.1.jar           3.69MB byte[] for java.lang.String
   1.91MB c.f.jackson.core.jackson-databind-2.15.3.jar         3.62MB java.lang.String
   1.20MB io.fabric8.openshift-model-6.8.1.jar                 3.32MB byte[] for general heap data
   1.16MB modified-io.vertx.vertx-core-4.4.6.jar               2.58MB byte[] for reflection metadata
 802.50kB q.0.0-SNAPSHOT-runner.jar                            1.26MB com.oracle.svm.core.hub.DynamicHubCompanion
 562.71kB com.fasterxml.jackson.core.jackson-core-2.15.3.jar   1.21MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
 556.74kB org.yaml.snakeyaml-2.1.jar                         749.27kB java.lang.String[]
 495.56kB org.snakeyaml.snakeyaml-engine-2.6.jar             596.81kB java.util.HashMap$Node
   8.92MB for 116 more packages                                5.76MB for 3308 more object types

Quarkus main

[8/8] Creating image...       [***]                                                                      (6.9s @ 4.76GB)
  78.78MB (52.68%) for code area:   174,296 compilation units
  70.37MB (47.06%) for image heap:  651,043 objects and 10 resources
 404.81kB ( 0.26%) for other data
 149.54MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  14.27MB io.fabric8.openshift-model-6.10.0.jar               16.53MB byte[] for code metadata
  12.87MB java.base                                            9.47MB byte[] for java.lang.String
   9.10MB io.fabric8.kubernetes-model-core-6.10.0.jar          8.98MB byte[] for reflection metadata
   8.79MB svm.jar (Native Image)                               8.14MB java.lang.Class
   2.90MB i.f.k.10.0.jar                                       5.84MB java.lang.String
   2.80MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar    5.14MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   2.43MB io.fabric8.kubernetes-model-flowcontrol-6.10.0.jar   4.77MB byte[] for general heap data
   1.95MB c.f.jackson.core.jackson-databind-2.16.1.jar         1.82MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.72MB i.f.kubernetes-model-apiextensions-6.10.0.jar        1.29MB java.lang.Object[]
   1.71MB io.fabric8.kubernetes-model-autoscaling-6.10.0.jar   1.20MB java.lang.String[]
  18.90MB for 116 more packages                                6.53MB for 3331 more object types

Quarkus main patched with diff from #38683 (comment)

[8/8] Creating image...       [**]                                                                       (4.5s @ 5.79GB)
  52.73MB (54.21%) for code area:   100,164 compilation units
  44.14MB (45.38%) for image heap:  463,734 objects and 10 resources
 402.77kB ( 0.40%) for other data
  97.27MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  12.86MB java.base                                           10.91MB byte[] for code metadata
   6.71MB io.fabric8.openshift-model-6.10.0.jar                5.47MB byte[] for java.lang.String
   5.46MB svm.jar (Native Image)                               4.61MB java.lang.Class
   4.30MB io.fabric8.kubernetes-model-core-6.10.0.jar          4.29MB java.lang.String
   1.95MB c.f.jackson.core.jackson-databind-2.16.1.jar         3.76MB byte[] for general heap data
   1.44MB i.f.k.10.0.jar                                       3.42MB byte[] for reflection metadata
   1.31MB io.fabric8.kubernetes-model-gatewayapi-6.10.0.jar    1.59MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   1.24MB io.fabric8.kubernetes-model-flowcontrol-6.10.0.jar   1.50MB com.oracle.svm.core.hub.DynamicHubCompanion
   1.18MB modified-io.vertx.vertx-core-4.5.3.jar             880.38kB java.lang.String[]
 997.95kB q.0.0-SNAPSHOT-runner.jar                          679.41kB java.lang.Object[]
  14.54MB for 116 more packages                                6.06MB for 3318 more object types

zakkak added a commit to zakkak/quarkus that referenced this issue Feb 20, 2024
`ValidationSchema` is annotated with `@JsonDeserialize` which leads in
its entire type hierarchy being registered for reflective access along
with the corresponding methods. This essentially ends up registering all
models as in kubernetes-client 6.9.0 ValidationSchema was augmented to
implement `Editable<ValidationSchemaBuilder>`, which increases the
reachable types in comparison to previous versions.

Ignoring registrations for `ValidationSchema` aligns with what we
already do for `KubeSchema`.

Fixes quarkusio#38683
@quarkus-bot quarkus-bot bot added this to the 3.9 - main milestone Feb 20, 2024
@gsmet gsmet modified the milestones: 3.9 - main, 3.7.4 Feb 20, 2024
gsmet pushed a commit to gsmet/quarkus that referenced this issue Feb 20, 2024
`ValidationSchema` is annotated with `@JsonDeserialize` which leads in
its entire type hierarchy being registered for reflective access along
with the corresponding methods. This essentially ends up registering all
models as in kubernetes-client 6.9.0 ValidationSchema was augmented to
implement `Editable<ValidationSchemaBuilder>`, which increases the
reachable types in comparison to previous versions.

Ignoring registrations for `ValidationSchema` aligns with what we
already do for `KubeSchema`.

Fixes quarkusio#38683

(cherry picked from commit 637ebea)
gsmet pushed a commit to gsmet/quarkus that referenced this issue Feb 20, 2024
`ValidationSchema` is annotated with `@JsonDeserialize` which leads in
its entire type hierarchy being registered for reflective access along
with the corresponding methods. This essentially ends up registering all
models as in kubernetes-client 6.9.0 ValidationSchema was augmented to
implement `Editable<ValidationSchemaBuilder>`, which increases the
reachable types in comparison to previous versions.

Ignoring registrations for `ValidationSchema` aligns with what we
already do for `KubeSchema`.

Fixes quarkusio#38683

(cherry picked from commit 637ebea)
benkard added a commit to benkard/mulkcms2 that referenced this issue Apr 6, 2024
This MR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [flow-bin](https://github.com/flowtype/flow-bin) ([changelog](https://github.com/facebook/flow/blob/master/Changelog.md)) | devDependencies | minor | [`^0.225.0` -> `^0.233.0`](https://renovatebot.com/diffs/npm/flow-bin/0.225.1/0.233.0) |
| [org.postgresql:postgresql](https://jdbc.postgresql.org) ([source](https://github.com/pgjdbc/pgjdbc)) | build | patch | `42.7.1` -> `42.7.3` |
| [org.liquibase.ext:liquibase-hibernate5](https://github.com/liquibase/liquibase-hibernate/wiki) ([source](https://github.com/liquibase/liquibase-hibernate)) | build | minor | `4.25.1` -> `4.27.0` |
| [org.liquibase:liquibase-maven-plugin](http://www.liquibase.org/liquibase-maven-plugin) ([source](https://github.com/liquibase/liquibase)) | build | minor | `4.25.1` -> `4.27.0` |
| [io.hypersistence:hypersistence-utils-hibernate-62](https://github.com/vladmihalcea/hypersistence-utils) | compile | patch | `3.7.0` -> `3.7.3` |
| [org.hibernate.orm:hibernate-envers](https://hibernate.org/orm) ([source](https://github.com/hibernate/hibernate-orm)) | build | patch | `6.4.1.Final` -> `6.4.4.Final` |
| [org.hibernate.orm:hibernate-core](https://hibernate.org/orm) ([source](https://github.com/hibernate/hibernate-orm)) | build | patch | `6.4.1.Final` -> `6.4.4.Final` |
| [com.blazebit:blaze-persistence-bom](https://persistence.blazebit.com) ([source](https://github.com/Blazebit/blaze-persistence)) | import | patch | `1.6.10` -> `1.6.11` |
| [com.diffplug.spotless:spotless-maven-plugin](https://github.com/diffplug/spotless) | build | minor | `2.41.1` -> `2.43.0` |
| [io.quarkus:quarkus-maven-plugin](https://github.com/quarkusio/quarkus) | build | minor | `3.6.4` -> `3.9.2` |
| [io.quarkus:quarkus-universe-bom](https://github.com/quarkusio/quarkus-platform) | import | minor | `3.6.4` -> `3.9.2` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://maven.apache.org/plugins/) | build | minor | `3.12.1` -> `3.13.0` |

---

### Release Notes

<details>
<summary>flowtype/flow-bin</summary>

### [`v0.233.0`](https://github.com/flowtype/flow-bin/compare/6e34f048ec7f5146297e258a60250c8e5af37bcc...2ebcdf3a8f03993e8ccab9e9fb6742000b54f929)

[Compare Source](https://github.com/flowtype/flow-bin/compare/6e34f048ec7f5146297e258a60250c8e5af37bcc...2ebcdf3a8f03993e8ccab9e9fb6742000b54f929)

### [`v0.232.0`](https://github.com/flowtype/flow-bin/compare/69ee58d99676a48984158d2cafcdb3b3f5ad5f15...6e34f048ec7f5146297e258a60250c8e5af37bcc)

[Compare Source](https://github.com/flowtype/flow-bin/compare/69ee58d99676a48984158d2cafcdb3b3f5ad5f15...6e34f048ec7f5146297e258a60250c8e5af37bcc)

### [`v0.231.0`](https://github.com/flowtype/flow-bin/compare/5c84049e450b37833fca5b547c1c2cb678436ef1...69ee58d99676a48984158d2cafcdb3b3f5ad5f15)

[Compare Source](https://github.com/flowtype/flow-bin/compare/5c84049e450b37833fca5b547c1c2cb678436ef1...69ee58d99676a48984158d2cafcdb3b3f5ad5f15)

### [`v0.230.0`](https://github.com/flowtype/flow-bin/compare/2c3181fa7aa928bd3735a7fad09e1be271c96c95...5c84049e450b37833fca5b547c1c2cb678436ef1)

[Compare Source](https://github.com/flowtype/flow-bin/compare/2c3181fa7aa928bd3735a7fad09e1be271c96c95...5c84049e450b37833fca5b547c1c2cb678436ef1)

### [`v0.229.2`](https://github.com/flowtype/flow-bin/compare/82b999003b85e827cd4dd36a8d3593979f1a9599...2c3181fa7aa928bd3735a7fad09e1be271c96c95)

[Compare Source](https://github.com/flowtype/flow-bin/compare/82b999003b85e827cd4dd36a8d3593979f1a9599...2c3181fa7aa928bd3735a7fad09e1be271c96c95)

### [`v0.229.0`](https://github.com/flowtype/flow-bin/compare/3d62fc76bf9b0ff63ec56d049c669958ef41f6b8...82b999003b85e827cd4dd36a8d3593979f1a9599)

[Compare Source](https://github.com/flowtype/flow-bin/compare/3d62fc76bf9b0ff63ec56d049c669958ef41f6b8...82b999003b85e827cd4dd36a8d3593979f1a9599)

### [`v0.228.0`](https://github.com/flowtype/flow-bin/compare/15db2846c1c63d3f26905f51e8c96c801cbc017b...3d62fc76bf9b0ff63ec56d049c669958ef41f6b8)

[Compare Source](https://github.com/flowtype/flow-bin/compare/15db2846c1c63d3f26905f51e8c96c801cbc017b...3d62fc76bf9b0ff63ec56d049c669958ef41f6b8)

### [`v0.227.0`](https://github.com/flowtype/flow-bin/compare/6fbe6faecdcb24e9ee660a0616705d46b9bd3c40...15db2846c1c63d3f26905f51e8c96c801cbc017b)

[Compare Source](https://github.com/flowtype/flow-bin/compare/6fbe6faecdcb24e9ee660a0616705d46b9bd3c40...15db2846c1c63d3f26905f51e8c96c801cbc017b)

### [`v0.226.0`](https://github.com/flowtype/flow-bin/compare/23ec6163cf6921d4ef74da53e1aaf4a35f798384...6fbe6faecdcb24e9ee660a0616705d46b9bd3c40)

[Compare Source](https://github.com/flowtype/flow-bin/compare/23ec6163cf6921d4ef74da53e1aaf4a35f798384...6fbe6faecdcb24e9ee660a0616705d46b9bd3c40)

</details>

<details>
<summary>pgjdbc/pgjdbc</summary>

### [`v42.7.3`](https://github.com/pgjdbc/pgjdbc/blob/HEAD/CHANGELOG.md#&#8203;4273-2024-04-14-145100--0400)

##### Changed

-   chore: gradle config enforces 17+ [MR #&#8203;3147](https://github.com/pgjdbc/pgjdbc/pull/3147)

##### Fixed

-   fix: boolean types not handled in SimpleQuery mode [MR #&#8203;3146](https://github.com/pgjdbc/pgjdbc/pull/3146)
    -   make sure we handle boolean types in simple query mode
    -   support uuid as well
    -   handle all well known types in text mode and change `else if` to `switch`
-   fix: released new versions of 42.2.29, 42.3.10, 42.4.5, 42.5.6, 42.6.2 to deal with `NoSuchMethodError on ByteBuffer#position` when running on Java 8

### [`v42.7.2`](https://github.com/pgjdbc/pgjdbc/blob/HEAD/CHANGELOG.md#&#8203;4272-2024-02-21-082300--0500)

##### Security

-   security: SQL Injection via line comment generation, it is possible in `SimpleQuery` mode to generate a line comment by having a placeholder for a numeric with a `-`
    such as `-?`. There must be second placeholder for a string immediately after. Setting the parameter to a -ve value creates a line comment.
    This has been fixed in this version fixes [CVE-2024-1597](https://www.cve.org/CVERecord?id=CVE-2024-1597). Reported by [Paul Gerste](https://github.com/paul-gerste-sonarsource). See the [security advisory](https://github.com/pgjdbc/pgjdbc/security/advisories/GHSA-24rp-q3w6-vc56) for more details. This has been fixed in versions 42.7.2, 42.6.1 42.5.5, 42.4.4, 42.3.9, 42.2.28.jre7. See the security advisory for work arounds.

##### Changed

-   fix: Use simple query for isValid. Using Extended query sends two messages checkConnectionQuery was never ever set or used, removed [MR #&#8203;3101](https://github.com/pgjdbc/pgjdbc/pull/3101)
-   perf: Avoid autoboxing bind indexes by [@&#8203;bokken](https://github.com/bokken) in [MR #&#8203;1244](https://github.com/pgjdbc/pgjdbc/pull/1244)
-   refactor: Document that encodePassword will zero out the password array, and remove driver's default encodePassword by [@&#8203;vlsi](https://github.com/vlsi) in [MR #&#8203;3084](https://github.com/pgjdbc/pgjdbc/pull/3084)

##### Added

-   feat: Add PasswordUtil for encrypting passwords client side [MR #&#8203;3082](https://github.com/pgjdbc/pgjdbc/pull/3082)

</details>

<details>
<summary>liquibase/liquibase</summary>

### [`v4.27.0`](https://github.com/liquibase/liquibase/blob/HEAD/changelog.txt#Liquibase-4270-is-a-major-release)

[Compare Source](https://github.com/liquibase/liquibase/compare/v4.26.0...v4.27.0)

> Liquibase 4.27.0 contains several New Capabilities and Notable Enhancements for Liquibase Pro users: DATABASECHANGELOGHISTORY table, Quality Checks Chains, Rollback Reports
> See the [Liquibase 4.27.0 Release Notes](https://docs.liquibase.com/start/release-notes/liquibase-release-notes/liquibase-4.27.0.html) for the complete set of release information.

### [`v4.26.0`](https://github.com/liquibase/liquibase/blob/HEAD/changelog.txt#Liquibase-4260-is-a-major-release)

[Compare Source](https://github.com/liquibase/liquibase/compare/v4.25.1...v4.26.0)

> \[!IMPORTANT]
> Liquibase 4.26.0 contains several Notable Changes for Liquibase Pro users: Advanced IF conditionals, Simpler Regex-based pattern checks, and Checks Run Reports.

> \[!NOTE]
> See the [Liquibase 4.26.0 Release Notes](https://docs.liquibase.com/start/release-notes/liquibase-4.26.0.html) for the complete set of release information.

</details>

<details>
<summary>vladmihalcea/hypersistence-utils</summary>

### [`v3.7.3`](https://github.com/vladmihalcea/hypersistence-utils/blob/HEAD/changelog.txt#Version-373---February-16-2024)

\================================================================================

"java.lang.ClassCastException: class \[Ljava.lang.String; cannot be cast to class \[B" thrown when using multiLoad with Hibernate 6.4 [#&#8203;700](https://github.com/vladmihalcea/hypersistence-utils/issues/700)

### [`v3.7.2`](https://github.com/vladmihalcea/hypersistence-utils/blob/HEAD/changelog.txt#Version-372---February-08-2024)

\================================================================================

Expecting BasicPluralJavaType for array class \[Ljava.util.UUID;,
but got \`com.vladmihalcea.hibernate.type.array.internal.UUIDArrayTypeDescriptor error with Hibernate 6.4 [#&#8203;698](https://github.com/vladmihalcea/hypersistence-utils/issues/698)

### [`v3.7.1`](https://github.com/vladmihalcea/hypersistence-utils/blob/HEAD/changelog.txt#Version-371---January-30-2024)

\================================================================================

Add support for Hibernate 6.4 [#&#8203;685](https://github.com/vladmihalcea/hypersistence-utils/issues/685)

Remove hypersistence-utils-hibernate-5 module [#&#8203;693](https://github.com/vladmihalcea/hypersistence-utils/issues/693)

</details>

<details>
<summary>hibernate/hibernate-orm</summary>

### [`v6.4.4.Final`](https://github.com/hibernate/hibernate-orm/compare/6.4.3...6.4.4)

[Compare Source](https://github.com/hibernate/hibernate-orm/compare/6.4.3...6.4.4)

### [`v6.4.3.Final`](https://github.com/hibernate/hibernate-orm/compare/6.4.2...6.4.3)

[Compare Source](https://github.com/hibernate/hibernate-orm/compare/6.4.2...6.4.3)

### [`v6.4.2.Final`](https://github.com/hibernate/hibernate-orm/compare/6.4.1...6.4.2)

[Compare Source](https://github.com/hibernate/hibernate-orm/compare/6.4.1...6.4.2)

</details>

<details>
<summary>Blazebit/blaze-persistence</summary>

### [`v1.6.11`](https://github.com/Blazebit/blaze-persistence/blob/HEAD/CHANGELOG.md#&#8203;1611)

[Compare Source](https://github.com/Blazebit/blaze-persistence/compare/1.6.10...1.6.11)

10/01/2024 - [Release tag](https://github.com/Blazebit/blaze-persistence/releases/tag/1.6.11) [Resolved issues](https://github.com/Blazebit/blaze-persistence/issues?q=is%3Aissue+milestone%3A1.6.11+is%3Aclosed+sort%3Aupdated-desc)

##### New features

-   Special case mappings with limit of 1 to use `=` instead of `IN` predicate
-   Added support for extended GraphlQL types, for example support DateTime (please read `Backwards-incompatible changes` below )

##### Bug fixes

-   Fix over-fetching of entity view data with dynamic fetches
-   Fix Spring 6.1 compatibility
-   Fix Hibernate 6.4.0.Final compatibility
-   Disallow repository sorting by anything other than entity or entity view attribute paths

##### Backwards-incompatible changes

-   If you use the GraphQL integration and you are loading the [graphql-java-extended-scalars](https://github.com/graphql-java/graphql-java-extended-scalars), you might need to migrate your `LocalDataTime` EntityViews to `OffsetDateTime`. Dates will no longer be represented as String in the GraphQl-Schema, but as [DateTime](https://the-guild.dev/graphql/scalars/docs/scalars/date-time).

</details>

<details>
<summary>diffplug/spotless</summary>

### [`v2.43.0`](https://github.com/diffplug/spotless/blob/HEAD/CHANGES.md#&#8203;2430---2023-11-27)

##### Added

-   Support custom rule sets for Ktlint. ([#&#8203;1896](https://github.com/diffplug/spotless/pull/1896))

##### Fixed

-   Fix Eclipse JDT on some settings files. ([#&#8203;1864](https://github.com/diffplug/spotless/pull/1864) fixes [#&#8203;1638](https://github.com/diffplug/spotless/issues/1638))

##### Changes

-   Bump default `ktlint` version to latest `1.0.0` -> `1.0.1`. ([#&#8203;1855](https://github.com/diffplug/spotless/pull/1855))
-   Add a Step to remove semicolons from Groovy files. ([#&#8203;1881](https://github.com/diffplug/spotless/pull/1881))

### [`v2.42.0`](https://github.com/diffplug/spotless/blob/HEAD/CHANGES.md#&#8203;2420---2023-09-28)

##### Added

-   Support for biome. The Rome project [was renamed to Biome](https://biomejs.dev/blog/annoucing-biome/).
    The configuration is still the same, but you should switch to the new `biome` tag / function and adjust
    the version accordingly. ([#&#8203;1804](https://github.com/diffplug/spotless/issues/1804)).
-   Support for `google-java-format`'s `skip-javadoc-formatting` option. ([#&#8203;1793](https://github.com/diffplug/spotless/pull/1793))
-   Support configuration of mirrors for P2 repositories in Maven DSL ([#&#8203;1697](https://github.com/diffplug/spotless/issues/1697)).
-   New line endings mode `GIT_ATTRIBUTES_FAST_ALLSAME`. ([#&#8203;1838](https://github.com/diffplug/spotless/pull/1838))

##### Fixed

-   Fix support for plugins when using Prettier version `3.0.0` and newer. ([#&#8203;1802](https://github.com/diffplug/spotless/pull/1802))
-   Fix configuration cache issue around `external process started '/usr/bin/git --version'`. ([#&#8203;1806](https://github.com/diffplug/spotless/issues/1806))

##### Changes

-   Bump default `flexmark` version to latest `0.64.0` -> `0.64.8`. ([#&#8203;1801](https://github.com/diffplug/spotless/pull/1801))
-   Bump default `ktlint` version to latest `0.50.0` -> `1.0.0`. ([#&#8203;1808](https://github.com/diffplug/spotless/pull/1808))

</details>

<details>
<summary>quarkusio/quarkus</summary>

### [`v3.9.2`](https://github.com/quarkusio/quarkus/releases/tag/3.9.2)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.9.1...3.9.2)

##### Complete changelog

-   [#&#8203;38964](https://github.com/quarkusio/quarkus/pull/38964) - Add smallrye metrics capability
-   [#&#8203;39668](https://github.com/quarkusio/quarkus/pull/39668) - Recompute cache when the redis connection pool is exhausted
-   [#&#8203;39705](https://github.com/quarkusio/quarkus/pull/39705) - WebSockets Next: error handlers part 1
-   [#&#8203;39717](https://github.com/quarkusio/quarkus/issues/39717) - OIDC code flow access token verification is enforced even if the application code does not use it as JWT
-   [#&#8203;39718](https://github.com/quarkusio/quarkus/pull/39718) - Enforce OIDC code flow access token verification only if JWT is in the application code
-   [#&#8203;39725](https://github.com/quarkusio/quarkus/pull/39725) - Setting the correct CodeFlowVerifyAccessTokenDisabledTest test class name
-   [#&#8203;39742](https://github.com/quarkusio/quarkus/pull/39742) - Fix a remaining reference to quarkus.resteasy-reactive. prefix
-   [#&#8203;39746](https://github.com/quarkusio/quarkus/issues/39746) - ScheduledExecutorService: cannot remove future task from the scheduler
-   [#&#8203;39763](https://github.com/quarkusio/quarkus/pull/39763) - WebSockets Next: error handlers part 2
-   [#&#8203;39766](https://github.com/quarkusio/quarkus/pull/39766) - Properly handle array class types to be looked up
-   [#&#8203;39770](https://github.com/quarkusio/quarkus/pull/39770) - Improve the multipart encoded mode handling in the rest client
-   [#&#8203;39776](https://github.com/quarkusio/quarkus/issues/39776) - Cannot build native image after 3.9.1 upgrade - missing configuration properties but they exist
-   [#&#8203;39777](https://github.com/quarkusio/quarkus/issues/39777) - Bean Param init issue -  java.lang.NoClassDefFoundError: io/quarkus/generated/int$quarkusrestparamConverter$
-   [#&#8203;39790](https://github.com/quarkusio/quarkus/pull/39790) - Add quarkus-credentials-deployment where it's missing
-   [#&#8203;39794](https://github.com/quarkusio/quarkus/pull/39794) - Bump org.jboss.threads:jboss-threads from 3.6.0.Final to 3.6.1.Final
-   [#&#8203;39797](https://github.com/quarkusio/quarkus/pull/39797) - Docs: fix typo in OIDC tenant resolution by configuration
-   [#&#8203;39798](https://github.com/quarkusio/quarkus/issues/39798) - Update documentation for QUARKUS AND GRADLE - Development mode: quarkusDev#workingDir has been deprecated
-   [#&#8203;39804](https://github.com/quarkusio/quarkus/pull/39804) - Always record profiles
-   [#&#8203;39823](https://github.com/quarkusio/quarkus/pull/39823) - Ensure ParameterConverter is loaded from the TCCL
-   [#&#8203;39829](https://github.com/quarkusio/quarkus/pull/39829) - Use quarkusDev#workingDirectory
-   [#&#8203;39835](https://github.com/quarkusio/quarkus/pull/39835) - Bump maven-model-helper to 36

### [`v3.9.1`](https://github.com/quarkusio/quarkus/releases/tag/3.9.1)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.9.0...3.9.1)

##### Complete changelog

-   [#&#8203;25682](https://github.com/quarkusio/quarkus/issues/25682) - Dev Services for Postgresql not working with Rancher
-   [#&#8203;36736](https://github.com/quarkusio/quarkus/pull/36736) - Manage Jose4j dependency in the bom
-   [#&#8203;36737](https://github.com/quarkusio/quarkus/issues/36737) - Add OpenAPI Filter usage to documentation
-   [#&#8203;39088](https://github.com/quarkusio/quarkus/issues/39088) - Solve POM formatting issues when creating project/adding extension/removing extension
-   [#&#8203;39224](https://github.com/quarkusio/quarkus/issues/39224) - WebSockets Next: support method parameter injection
-   [#&#8203;39313](https://github.com/quarkusio/quarkus/issues/39313) - prod-profile configuration pollutes test profile in integration tests
-   [#&#8203;39371](https://github.com/quarkusio/quarkus/pull/39371) - Avoid resolving plugin command after the first option
-   [#&#8203;39382](https://github.com/quarkusio/quarkus/pull/39382) - Preserve POM format when extensions are added/removed
-   [#&#8203;39385](https://github.com/quarkusio/quarkus/issues/39385) - smallrye-health should hande the case when Vert.x current context is null
-   [#&#8203;39388](https://github.com/quarkusio/quarkus/issues/39388) - Hibernate runtime property persisting after build
-   [#&#8203;39394](https://github.com/quarkusio/quarkus/pull/39394) - Handle null Vert.x context in smallrye-health
-   [#&#8203;39426](https://github.com/quarkusio/quarkus/pull/39426) - Do not record active profile configuration name if a profile one exists
-   [#&#8203;39443](https://github.com/quarkusio/quarkus/pull/39443) - Use Quarkus wide version of jna-platform in azure-functions
-   [#&#8203;39496](https://github.com/quarkusio/quarkus/issues/39496) - Gradle build cache prevents source packages to be installed to local Maven repository
-   [#&#8203;39513](https://github.com/quarkusio/quarkus/issues/39513) - `@SecureField` in members of the response class isn't applied
-   [#&#8203;39528](https://github.com/quarkusio/quarkus/issues/39528) - (Doc issue)  Getting token using blocking or non blocking calls
-   [#&#8203;39544](https://github.com/quarkusio/quarkus/issues/39544) - OidcClient: Getting exception when trying to use tokenHelper
-   [#&#8203;39546](https://github.com/quarkusio/quarkus/issues/39546) - Make maxParameters of MultiPartUploadHandler configurable
-   [#&#8203;39549](https://github.com/quarkusio/quarkus/pull/39549) - Make max parameters of multipart handling configurable
-   [#&#8203;39564](https://github.com/quarkusio/quarkus/pull/39564) - Fix Quarkus REST Jackson `@SecureField` detection on subclasses, interface implementors, fileds of the fields, parametrized types and arrays
-   [#&#8203;39572](https://github.com/quarkusio/quarkus/pull/39572) - Remove mvnpm and webjars from the 404 page
-   [#&#8203;39574](https://github.com/quarkusio/quarkus/pull/39574) - Add a section about openapi filters in the doc
-   [#&#8203;39576](https://github.com/quarkusio/quarkus/pull/39576) - Fix semconv-stability.opt-in property name
-   [#&#8203;39578](https://github.com/quarkusio/quarkus/pull/39578) - Update quarkus-project-develocity-extension to 1.0.7
-   [#&#8203;39579](https://github.com/quarkusio/quarkus/pull/39579) - Don't run CDI interceptors on class-level exception mappers
-   [#&#8203;39580](https://github.com/quarkusio/quarkus/pull/39580) - Fix directory name in vertx.adoc
-   [#&#8203;39581](https://github.com/quarkusio/quarkus/issues/39581) - The http metrics provide a path instead of REDIRECTION and NOT_FOUND when possible
-   [#&#8203;39583](https://github.com/quarkusio/quarkus/pull/39583) - Keep the URIs in the metrics tag if they match a client or server pattern
-   [#&#8203;39586](https://github.com/quarkusio/quarkus/issues/39586) - RestMulti is not sending headers if there is no content
-   [#&#8203;39587](https://github.com/quarkusio/quarkus/pull/39587) - Properly use headers from RestMulti when the multi is empty
-   [#&#8203;39589](https://github.com/quarkusio/quarkus/pull/39589) - Only Add OTel Security Events when span is recording
-   [#&#8203;39594](https://github.com/quarkusio/quarkus/pull/39594) - Redis: add documentation for replicas usage
-   [#&#8203;39595](https://github.com/quarkusio/quarkus/pull/39595) - Update one of OIDC certificate chain tests to use TenantConfigResolver
-   [#&#8203;39598](https://github.com/quarkusio/quarkus/issues/39598) - ClassNotFoundException for beanparam class with generics in external artifact
-   [#&#8203;39599](https://github.com/quarkusio/quarkus/issues/39599) - JsonObject is empty when used with resteasy-reactive
-   [#&#8203;39604](https://github.com/quarkusio/quarkus/pull/39604) - Do not record local sources in runtime config defaults.
-   [#&#8203;39615](https://github.com/quarkusio/quarkus/pull/39615) - Allow JsonObject and JsonArray to be used in any POJO for JSON handling
-   [#&#8203;39623](https://github.com/quarkusio/quarkus/issues/39623) - Type not consistent in sample code
-   [#&#8203;39626](https://github.com/quarkusio/quarkus/pull/39626) - Update parameter type to be consistent across the doc
-   [#&#8203;39628](https://github.com/quarkusio/quarkus/pull/39628) - Bump smallrye-jwt.version from 4.4.0 to 4.5.0
-   [#&#8203;39630](https://github.com/quarkusio/quarkus/pull/39630) - Bump smallrye-reactive-messaging.version from 4.18.0 to 4.19.0
-   [#&#8203;39638](https://github.com/quarkusio/quarkus/pull/39638) - Avoid all caching in DevModeClient
-   [#&#8203;39642](https://github.com/quarkusio/quarkus/pull/39642) - WebSocket Next: endpoint callback arguments injection
-   [#&#8203;39645](https://github.com/quarkusio/quarkus/pull/39645) - Improve the OIDC Client Quickstart document
-   [#&#8203;39651](https://github.com/quarkusio/quarkus/pull/39651) - Bump io.quarkus.bot:build-reporter-maven-extension from 3.5.0 to 3.6.0
-   [#&#8203;39656](https://github.com/quarkusio/quarkus/pull/39656) - Bump maven-model-helper to 35
-   [#&#8203;39661](https://github.com/quarkusio/quarkus/pull/39661) - Fix property name in OIDC docs
-   [#&#8203;39679](https://github.com/quarkusio/quarkus/pull/39679) - Bump mime4j version to 0.8.11
-   [#&#8203;39682](https://github.com/quarkusio/quarkus/pull/39682) - Fix postgres datasource devservice not working with rancher-desktop on mac arm
-   [#&#8203;39691](https://github.com/quarkusio/quarkus/pull/39691) - Fix dev-mode issue with generated classes for Quarkus REST converters
-   [#&#8203;39699](https://github.com/quarkusio/quarkus/issues/39699) - UpxCompressionBuildStep Not Executed Since Quarkus 3.8.2
-   [#&#8203;39702](https://github.com/quarkusio/quarkus/pull/39702) - Bring back erroneously removed `@BuildStep`
-   [#&#8203;39706](https://github.com/quarkusio/quarkus/pull/39706) - Use --no-daemon when calling gradle update

### [`v3.9.0`](https://github.com/quarkusio/quarkus/releases/tag/3.9.0)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.8.3...3.9.0)

##### Complete changelog

-   [#&#8203;25101](https://github.com/quarkusio/quarkus/issues/25101) - \[CI] - quarkus-devtools-compat + Quarkus main
-   [#&#8203;27374](https://github.com/quarkusio/quarkus/issues/27374) - Update Quarkus CLI doc for installing specific version of the CLI to avoid printing warnings
-   [#&#8203;39144](https://github.com/quarkusio/quarkus/issues/39144) - WebSockets Next: documentation
-   [#&#8203;39315](https://github.com/quarkusio/quarkus/issues/39315) - `@ConfigMapping` handling of Maps is not compatible with old config classes
-   [#&#8203;39344](https://github.com/quarkusio/quarkus/issues/39344) - Avro schemas aren't generated in isolation
-   [#&#8203;39345](https://github.com/quarkusio/quarkus/pull/39345) - Isolate Avro schema code generation when using multiple schema files
-   [#&#8203;39363](https://github.com/quarkusio/quarkus/pull/39363) - Initial version of the WebSocket Next documentation
-   [#&#8203;39413](https://github.com/quarkusio/quarkus/pull/39413) - Refresh documentation (and some tests) of the Hibernate Search + ORM extension
-   [#&#8203;39427](https://github.com/quarkusio/quarkus/issues/39427) - Enforce authorization code flow access token verification if `JsonWebToken` is injected
-   [#&#8203;39428](https://github.com/quarkusio/quarkus/issues/39428) - Enforce OIDC UserInfo acquisition if `UserInfo` is injected
-   [#&#8203;39441](https://github.com/quarkusio/quarkus/issues/39441) - RESTEasy Reactive dependency added to deployment classpath of nearly all Quarkus apps
-   [#&#8203;39445](https://github.com/quarkusio/quarkus/pull/39445) - Remove Quarkus REST deployment dependency from Vertx HTTP deployment
-   [#&#8203;39447](https://github.com/quarkusio/quarkus/pull/39447) - Save concat indy allocations on JarResource::getResourceURL
-   [#&#8203;39454](https://github.com/quarkusio/quarkus/pull/39454) - Bump org.jboss.threads:jboss-threads from 3.5.1.Final to 3.6.0.Final
-   [#&#8203;39458](https://github.com/quarkusio/quarkus/pull/39458) - Enforce OIDC UserInfo acquisition and authorization code flow access token verification if UserInfo and JsonWebToken beans are injected
-   [#&#8203;39467](https://github.com/quarkusio/quarkus/pull/39467) - Fix codestarts compatibility with older CLI
-   [#&#8203;39468](https://github.com/quarkusio/quarkus/issues/39468) - ChainBuildException - Cycle detected after [#&#8203;39352](https://github.com/quarkusio/quarkus/issues/39352) MR
-   [#&#8203;39470](https://github.com/quarkusio/quarkus/pull/39470) - Remove the old MetricBuildItem SPI
-   [#&#8203;39471](https://github.com/quarkusio/quarkus/pull/39471) - Update to Vert.x 4.5.5
-   [#&#8203;39472](https://github.com/quarkusio/quarkus/pull/39472) - Update SmallRye Config to 3.7.0
-   [#&#8203;39474](https://github.com/quarkusio/quarkus/pull/39474) - Use explicit jar reference instead of GAV to avoid duplicate log warning
-   [#&#8203;39476](https://github.com/quarkusio/quarkus/pull/39476) - Fix the broken link to the OIDC client reference doc
-   [#&#8203;39477](https://github.com/quarkusio/quarkus/pull/39477) - Adjust toggle names in OTel InstrumentBuildTimeConfig
-   [#&#8203;39479](https://github.com/quarkusio/quarkus/issues/39479) - 3.9.0.CR2: NoClassDefFoundError: io/quarkus/security/spi/runtime/SecurityEvent
-   [#&#8203;39480](https://github.com/quarkusio/quarkus/pull/39480) - Fix security spi dependency on OTel
-   [#&#8203;39487](https://github.com/quarkusio/quarkus/pull/39487) - Allow occasional pin events in ShouldNotPin
-   [#&#8203;39491](https://github.com/quarkusio/quarkus/pull/39491) - Ignore the split access and refresh token cookies for resolving the tenant
-   [#&#8203;39519](https://github.com/quarkusio/quarkus/issues/39519) - OpenTelemetry - respect proxy settings in VertxGrpcExporter and VertxHttpExporter
-   [#&#8203;39522](https://github.com/quarkusio/quarkus/pull/39522) - Fix typos in rest doc
-   [#&#8203;39530](https://github.com/quarkusio/quarkus/pull/39530) - Update profile section in building-native-image.adoc
-   [#&#8203;39531](https://github.com/quarkusio/quarkus/pull/39531) - Use SmallRye Commons Inet
-   [#&#8203;39533](https://github.com/quarkusio/quarkus/issues/39533) - Class loader leak in configuration
-   [#&#8203;39536](https://github.com/quarkusio/quarkus/pull/39536) - Bump org.postgresql:postgresql from 42.7.2 to 42.7.3
-   [#&#8203;39541](https://github.com/quarkusio/quarkus/pull/39541) - Update SmallRye Config to 3.7.1
-   [#&#8203;39543](https://github.com/quarkusio/quarkus/pull/39543) - Enable proxy configuration for OpenTelemetry exporters
-   [#&#8203;39562](https://github.com/quarkusio/quarkus/pull/39562) - Bump io.smallrye.config:smallrye-config-source-yaml from 3.7.0 to 3.7.1 in /devtools/gradle

### [`v3.8.3`](https://github.com/quarkusio/quarkus/releases/tag/3.8.3)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.8.2...3.8.3)

##### Complete changelog

-   [#&#8203;25453](https://github.com/quarkusio/quarkus/issues/25453) - Mutiny is not compatible with quarkus opentelemetry
-   [#&#8203;31497](https://github.com/quarkusio/quarkus/issues/31497) - Enabled micrometer.binder.http-server should also capture parameterized sub-resources
-   [#&#8203;39047](https://github.com/quarkusio/quarkus/issues/39047) - Reactive pg datasource with enabled health check opens more connections than configured
-   [#&#8203;39145](https://github.com/quarkusio/quarkus/issues/39145) - Hibernate schema validation is flaky and fails due missing tables (while the tables are present)
-   [#&#8203;39162](https://github.com/quarkusio/quarkus/pull/39162) - Add mapping to a Map\<String, ConfigObject> in the documentation
-   [#&#8203;39178](https://github.com/quarkusio/quarkus/pull/39178) - Update grpc-service-implementation.adoc
-   [#&#8203;39192](https://github.com/quarkusio/quarkus/pull/39192) - Make HTTP templates for observability work with subresources
-   [#&#8203;39197](https://github.com/quarkusio/quarkus/issues/39197) - Qute is not adding the right NativeImageResourceBuildItem when using a custom template root
-   [#&#8203;39204](https://github.com/quarkusio/quarkus/issues/39204) - Update partial extension names to include full extension names
-   [#&#8203;39216](https://github.com/quarkusio/quarkus/pull/39216) - Unwrap processing exception from REST Client when returning a Uni
-   [#&#8203;39223](https://github.com/quarkusio/quarkus/pull/39223) - Fix WithSpan uni and multi
-   [#&#8203;39225](https://github.com/quarkusio/quarkus/pull/39225) - Upgrade to Mutiny 2.5.8
-   [#&#8203;39242](https://github.com/quarkusio/quarkus/issues/39242) - e quarkus-azure-functions-http  generationg function.json with missing method.
-   [#&#8203;39245](https://github.com/quarkusio/quarkus/issues/39245) - dev-ui shows wrong property for rest-client
-   [#&#8203;39251](https://github.com/quarkusio/quarkus/pull/39251) - Make mutiny version of pool use the already configured vertx pool
-   [#&#8203;39252](https://github.com/quarkusio/quarkus/pull/39252) - Explicitly set all HTTP methods for Azure Functions
-   [#&#8203;39255](https://github.com/quarkusio/quarkus/pull/39255) - Fix config key for dev-ui
-   [#&#8203;39257](https://github.com/quarkusio/quarkus/issues/39257) - Quarkus 3.8.1: Use GraalVM sdk 23.1.2 over 23.0.1
-   [#&#8203;39260](https://github.com/quarkusio/quarkus/pull/39260) - Bump GraalVM SDK version to 23.1.2
-   [#&#8203;39265](https://github.com/quarkusio/quarkus/pull/39265) - Properly support sending InputStream in REST Client
-   [#&#8203;39266](https://github.com/quarkusio/quarkus/issues/39266) - ./mvnw --file $(pwd)/./pom.xml broken since quarkus 3.7.1
-   [#&#8203;39270](https://github.com/quarkusio/quarkus/issues/39270) - Update the title of dev-ui.adoc
-   [#&#8203;39271](https://github.com/quarkusio/quarkus/pull/39271) - Update the title of dev-ui.adoc and fix minor typos
-   [#&#8203;39294](https://github.com/quarkusio/quarkus/pull/39294) - Qute: add correct NativeImageResourceBuildItem for custom template root
-   [#&#8203;39309](https://github.com/quarkusio/quarkus/pull/39309) - Normalize POM path
-   [#&#8203;39310](https://github.com/quarkusio/quarkus/issues/39310) - Wrong reference on list of injected beans
-   [#&#8203;39311](https://github.com/quarkusio/quarkus/pull/39311) - Be more consistent with extension names in datasource.adoc
-   [#&#8203;39316](https://github.com/quarkusio/quarkus/issues/39316) - Empty container-group not allowed in Quarkus 3.8.2
-   [#&#8203;39319](https://github.com/quarkusio/quarkus/pull/39319) - org.graalvm.js:js was renamed to org.graalvm.polyglot:js-community
-   [#&#8203;39337](https://github.com/quarkusio/quarkus/pull/39337) - Fix rest-client-mutiny mention in the docs
-   [#&#8203;39350](https://github.com/quarkusio/quarkus/issues/39350) - Exception when building application with a lot of dependencies, String too large to record error
-   [#&#8203;39352](https://github.com/quarkusio/quarkus/pull/39352) - Make Hibernate / Micrometer integration run after schema creation
-   [#&#8203;39353](https://github.com/quarkusio/quarkus/pull/39353) - Allow config empty values in the Gradle worker
-   [#&#8203;39354](https://github.com/quarkusio/quarkus/pull/39354) - Add note about pre-matching filters execution model
-   [#&#8203;39368](https://github.com/quarkusio/quarkus/issues/39368) - Submodule on second level fails to find itself in dev mode
-   [#&#8203;39372](https://github.com/quarkusio/quarkus/pull/39372) - Replace `org.graalvm.sdk:graal-sdk` dependency with `org.graalvm.sdk:nativeimage`
-   [#&#8203;39379](https://github.com/quarkusio/quarkus/pull/39379) - Bump org.apache.commons:commons-compress from 1.26.0 to 1.26.1
-   [#&#8203;39383](https://github.com/quarkusio/quarkus/pull/39383) - Typo at OIDC Client Mutual TLS config properties
-   [#&#8203;39386](https://github.com/quarkusio/quarkus/pull/39386) - Update to Brotli4J 1.16.0
-   [#&#8203;39402](https://github.com/quarkusio/quarkus/pull/39402) - ArC: fix creation of synthetic beans
-   [#&#8203;39411](https://github.com/quarkusio/quarkus/pull/39411) - Fix typo in Building my first extension
-   [#&#8203;39418](https://github.com/quarkusio/quarkus/pull/39418) - Use the value of project/default-codestart from the platform descriptor as the default codestart instead of a hardcoded value
-   [#&#8203;39430](https://github.com/quarkusio/quarkus/pull/39430) - Fix misleading error message when REST Client interface has been indexed
-   [#&#8203;39434](https://github.com/quarkusio/quarkus/pull/39434) - Qute: fix the NoRestartTemplatesDevModeTest on Windows
-   [#&#8203;39437](https://github.com/quarkusio/quarkus/pull/39437) - Make sure the current project location isn't overridden by other modules with the same groupId and artifactId
-   [#&#8203;39440](https://github.com/quarkusio/quarkus/issues/39440) - graal-sdk in 23.1.x brings in `org.graalvm.polyglot` which causes a couple of issues (wrap up)
-   [#&#8203;39442](https://github.com/quarkusio/quarkus/pull/39442) - Exclude org.graalvm.polyglot:polyglot from graal-sdk

### [`v3.8.2`](https://github.com/quarkusio/quarkus/releases/tag/3.8.2)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.8.1...3.8.2)

##### Complete changelog

-   [#&#8203;19849](https://github.com/quarkusio/quarkus/issues/19849) - Reactive rest client invoke MessageBodyReader.isReadable with null value of annotations parameter
-   [#&#8203;27999](https://github.com/quarkusio/quarkus/issues/27999) - quarkus.datasource."datasource-name".jdbc.min-size not honored if max-lifetime is set
-   [#&#8203;35993](https://github.com/quarkusio/quarkus/issues/35993) - Event-loop thread blocked when connecting to an unavailable OIDC server
-   [#&#8203;37984](https://github.com/quarkusio/quarkus/issues/37984) - Custom SecretsKeyHandler not found after update to 3.6.4
-   [#&#8203;38007](https://github.com/quarkusio/quarkus/issues/38007) - Failure to resolve encrypted configuration properties with the Gradle plugin
-   [#&#8203;38392](https://github.com/quarkusio/quarkus/issues/38392) - Application.properties string substitution does not work when using gradle variables
-   [#&#8203;38424](https://github.com/quarkusio/quarkus/issues/38424) - application-test.yml is not utilized during tests executed during gradle build
-   [#&#8203;38435](https://github.com/quarkusio/quarkus/pull/38435) - Fix CLI not recognizing installed plugins
-   [#&#8203;38900](https://github.com/quarkusio/quarkus/pull/38900) - Bump Smallrye Reactive Messaging from 4.16.2 to 4.17.0
-   [#&#8203;38971](https://github.com/quarkusio/quarkus/pull/38971) - Clarify that `quarkus.profile` cannot be set from a profile aware file
-   [#&#8203;38988](https://github.com/quarkusio/quarkus/pull/38988) - Do not expand configuration for Gradle cache
-   [#&#8203;38989](https://github.com/quarkusio/quarkus/issues/38989) - cert chain public key resolver thumbprints
-   [#&#8203;39001](https://github.com/quarkusio/quarkus/pull/39001) - Update to Vert.x 4.5.4 and Netty 4.1.107
-   [#&#8203;39021](https://github.com/quarkusio/quarkus/pull/39021) - Upgrade to testcontainers 1.19.6
-   [#&#8203;39023](https://github.com/quarkusio/quarkus/pull/39023) - Remove selector field from generated Job manifest in docs
-   [#&#8203;39041](https://github.com/quarkusio/quarkus/issues/39041) - JAX-RS seeOther does not work with IPv6
-   [#&#8203;39046](https://github.com/quarkusio/quarkus/pull/39046) - Make sure Response and RestResponse work properly with IPv6 addresses
-   [#&#8203;39057](https://github.com/quarkusio/quarkus/pull/39057) - Skip analysis of plugin executions with phases post quarkus:dev preparing for dev mode launch
-   [#&#8203;39059](https://github.com/quarkusio/quarkus/issues/39059) - Exception in blocking graphql query is wrapped
-   [#&#8203;39063](https://github.com/quarkusio/quarkus/pull/39063) - Fix the OIDC token verification failure with the inlined cert chain
-   [#&#8203;39067](https://github.com/quarkusio/quarkus/pull/39067) - Updates to Infinispan 14.0.25.Final
-   [#&#8203;39068](https://github.com/quarkusio/quarkus/pull/39068) - Optionally run DNS lookup for OIDC server requests on worker thread
-   [#&#8203;39069](https://github.com/quarkusio/quarkus/pull/39069) - Do not fail UPX if compression level is not given
-   [#&#8203;39070](https://github.com/quarkusio/quarkus/pull/39070) - Doc: add Pulsar in Dev Services Overview
-   [#&#8203;39072](https://github.com/quarkusio/quarkus/pull/39072) - Update to Agroal 2.3
-   [#&#8203;39078](https://github.com/quarkusio/quarkus/pull/39078) - Unwrap actual GraphQL data fetching exception if it is wrapped
-   [#&#8203;39093](https://github.com/quarkusio/quarkus/pull/39093) - Fix cross-references in the Vert.x Reference Guide
-   [#&#8203;39094](https://github.com/quarkusio/quarkus/pull/39094) - Emphasize the need to add quarkus-junit5-mockito as a dependency to use mock injection
-   [#&#8203;39102](https://github.com/quarkusio/quarkus/pull/39102) - Properly pass annotations to MessageBodyReader in REST Client
-   [#&#8203;39120](https://github.com/quarkusio/quarkus/issues/39120) - Startup fails with Kafka Stream if topics for topics check not defined when check is disabled
-   [#&#8203;39121](https://github.com/quarkusio/quarkus/pull/39121) - Do not fail on resolve kafka streams topics when topics check disabled
-   [#&#8203;39122](https://github.com/quarkusio/quarkus/pull/39122) - Use bcrypt password mapper in elytron-security-jdbc docs
-   [#&#8203;39123](https://github.com/quarkusio/quarkus/issues/39123) - Quarkus Dev Services passes wrong volume path to Docker on Windows
-   [#&#8203;39130](https://github.com/quarkusio/quarkus/issues/39130) - When building images with jib the fast-jar-lib layer is always changed
-   [#&#8203;39136](https://github.com/quarkusio/quarkus/pull/39136) - Fix wrong volume host path being used on Windows
-   [#&#8203;39147](https://github.com/quarkusio/quarkus/pull/39147) - Keep the timestamps when copying jars and building JIB layers
-   [#&#8203;39160](https://github.com/quarkusio/quarkus/pull/39160) - Fail on conflicting deployment kinds
-   [#&#8203;39168](https://github.com/quarkusio/quarkus/pull/39168) - Remove misleading note from jacoco.enabled
-   [#&#8203;39169](https://github.com/quarkusio/quarkus/issues/39169) - Unable to produce multiple synthetic beans of same type having different identifiers
-   [#&#8203;39179](https://github.com/quarkusio/quarkus/pull/39179) - Allow setting the SettingsDecrypter when initializing a Maven artifact resolver
-   [#&#8203;39181](https://github.com/quarkusio/quarkus/pull/39181) - ArC: fix BeanConfiguratorBase#read()
-   [#&#8203;39201](https://github.com/quarkusio/quarkus/pull/39201) - Bump quarkus-http.version from 5.2.0.Final to 5.2.1.Final
-   [#&#8203;39203](https://github.com/quarkusio/quarkus/pull/39203) - Fix typo in testing Getting Started guide example

### [`v3.8.1`](https://github.com/quarkusio/quarkus/releases/tag/3.8.1)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.8.0...3.8.1)

##### Complete changelog

-   [#&#8203;5314](https://github.com/quarkusio/quarkus/issues/5314) - Subresouce init resource failed when using `ResourceContext.getResource`
-   [#&#8203;36427](https://github.com/quarkusio/quarkus/issues/36427) - Keycloak admin client fail with "authHeader" is null when using classic extensions
-   [#&#8203;37065](https://github.com/quarkusio/quarkus/issues/37065) - Azure Functions Http: missing HTTP method definitions for delete and patch
-   [#&#8203;37779](https://github.com/quarkusio/quarkus/issues/37779) - No healthcheck for default Agroal datasource if `quarkus.datasource.db-kind` is not set
-   [#&#8203;37962](https://github.com/quarkusio/quarkus/issues/37962) - Can't specify custom quarkus.profile when running tests
-   [#&#8203;38557](https://github.com/quarkusio/quarkus/issues/38557) - Overwriting application configuration does not work with .env File
-   [#&#8203;38798](https://github.com/quarkusio/quarkus/issues/38798) - Using custom header in REST client together with `@NotBody` annotated argument results in warning from EndpointIndexer
-   [#&#8203;38880](https://github.com/quarkusio/quarkus/issues/38880) - CronJob deployment doesn't work due to `selector` field
-   [#&#8203;38881](https://github.com/quarkusio/quarkus/pull/38881) - Remove selector field if it's empty from manifest
-   [#&#8203;38891](https://github.com/quarkusio/quarkus/pull/38891) - Reduce message log level
-   [#&#8203;38895](https://github.com/quarkusio/quarkus/pull/38895) - Make VertxGrpcExporter more robust
-   [#&#8203;38899](https://github.com/quarkusio/quarkus/pull/38899) - Fix Keycloak Admin Client Classic when used with the RESTEasy JSON-B and REST Client JSON-B extensions
-   [#&#8203;38901](https://github.com/quarkusio/quarkus/issues/38901) - OidcProvider throws NPE when certificate chain is configured with OIDC server which has no JWK keys at the startup
-   [#&#8203;38909](https://github.com/quarkusio/quarkus/pull/38909) - Bump org.postgresql:postgresql from 42.7.1 to 42.7.2
-   [#&#8203;38923](https://github.com/quarkusio/quarkus/pull/38923) - Allow all HTTP methods in Azure functions
-   [#&#8203;38925](https://github.com/quarkusio/quarkus/pull/38925) - Improve shutdown of VertxHttpExporter and VertxGrpcExporter
-   [#&#8203;38927](https://github.com/quarkusio/quarkus/pull/38927) - Use supplier in order to properly have mutiny retry
-   [#&#8203;38928](https://github.com/quarkusio/quarkus/issues/38928) - quarkus-quartz: CDIAwareJob destroys instance of Quartz Job too early when Job is a `@Dependent` bean
-   [#&#8203;38932](https://github.com/quarkusio/quarkus/pull/38932) - Fix NPE when OIDC token must be verified with the chain with OIDC server returning no JWKs
-   [#&#8203;38934](https://github.com/quarkusio/quarkus/issues/38934) - Agroal Data Source Health check failing for reactive data source
-   [#&#8203;38935](https://github.com/quarkusio/quarkus/pull/38935) - Upgrade to Mutiny 2.5.7
-   [#&#8203;38938](https://github.com/quarkusio/quarkus/pull/38938) - Propagate user.dir to Gradle worker
-   [#&#8203;38944](https://github.com/quarkusio/quarkus/pull/38944) - Bump smallrye-open-api.version from 3.9.0 to 3.10.0
-   [#&#8203;38949](https://github.com/quarkusio/quarkus/issues/38949) - Postgresql bump causing detection of instance Random/SplittableRandom
-   [#&#8203;38952](https://github.com/quarkusio/quarkus/issues/38952) - Properly pass errors from JsonRPC backends to Dev UI
-   [#&#8203;38953](https://github.com/quarkusio/quarkus/pull/38953) - Unwrap the actual failure from JsonRPC if it's wrapped
-   [#&#8203;38955](https://github.com/quarkusio/quarkus/pull/38955) - Try to get more disk space
-   [#&#8203;38957](https://github.com/quarkusio/quarkus/pull/38957) - Quartz - fix `@Dependent` job creation/destruction when there is a re-fire
-   [#&#8203;38958](https://github.com/quarkusio/quarkus/pull/38958) - Runtime reinitialize org.postgresql.util.PasswordUtil$SecureRandomHolder
-   [#&#8203;38959](https://github.com/quarkusio/quarkus/pull/38959) - Agroal - Only generate health checks for JDBC datasources
-   [#&#8203;38978](https://github.com/quarkusio/quarkus/pull/38978) - Bump org.mariadb.jdbc:mariadb-java-client from 3.3.2 to 3.3.3
-   [#&#8203;38979](https://github.com/quarkusio/quarkus/pull/38979) - Propagate quarkus.test.profile to Gradle worker
-   [#&#8203;38986](https://github.com/quarkusio/quarkus/pull/38986) - Add missing brace in property expression
-   [#&#8203;38990](https://github.com/quarkusio/quarkus/issues/38990) - Quarkus 3.7.4 java.lang.ClassNotFoundException when running devsevices with gradle
-   [#&#8203;38995](https://github.com/quarkusio/quarkus/pull/38995) - Take client methods into account in server endpoint indexer
-   [#&#8203;38997](https://github.com/quarkusio/quarkus/pull/38997) - Add hint about exporter collector protocol on generic gRPC error
-   [#&#8203;38999](https://github.com/quarkusio/quarkus/pull/38999) - Remove JetBrains `@Nullable` from RESTEasy Reactive code
-   [#&#8203;39006](https://github.com/quarkusio/quarkus/pull/39006) - Bump Keycloak version to 23.0.7
-   [#&#8203;39020](https://github.com/quarkusio/quarkus/pull/39020) - Make VertxHttpExporter more robust
-   [#&#8203;39022](https://github.com/quarkusio/quarkus/issues/39022) - `JAVA_APP_DIR` should be set for container images
-   [#&#8203;39024](https://github.com/quarkusio/quarkus/pull/39024) - Set JAVA_APP_DIR env var when necessary
-   [#&#8203;39028](https://github.com/quarkusio/quarkus/pull/39028) - Make Sub Resources unremovable beans
-   [#&#8203;39029](https://github.com/quarkusio/quarkus/pull/39029) - Update to Brotli 1.14.0
-   [#&#8203;39031](https://github.com/quarkusio/quarkus/pull/39031) - Add commons-codec to Dev Services dependencies

### [`v3.8.0`](https://github.com/quarkusio/quarkus/releases/tag/3.8.0)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.7.4...3.8.0)

##### Complete changelog

-   [#&#8203;35686](https://github.com/quarkusio/quarkus/issues/35686) - Sporadic "Failed to export spans. The request could not be executed. Full error message: Stream was closed"

### [`v3.7.4`](https://github.com/quarkusio/quarkus/releases/tag/3.7.4)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.7.3...3.7.4)

##### Complete changelog

-   [#&#8203;37608](https://github.com/quarkusio/quarkus/issues/37608) - gRPC starter app is using legacy approach, single HTTP server should be used instead
-   [#&#8203;38236](https://github.com/quarkusio/quarkus/issues/38236) - Adding a decorator causes bytecode error
-   [#&#8203;38504](https://github.com/quarkusio/quarkus/issues/38504) - NPE on oidc-client when quarkus.oidc-client.grant-options.password.password not provided
-   [#&#8203;38533](https://github.com/quarkusio/quarkus/issues/38533) - 'Unable to find a JDBC driver' for Hibernate Reactive after updating to 3.7.1
-   [#&#8203;38683](https://github.com/quarkusio/quarkus/issues/38683) - Build time performance regression and bigger native binaries when migrating from 3.5 to 3.6 or 3.7
-   [#&#8203;38688](https://github.com/quarkusio/quarkus/pull/38688) - Making sure deployment modules excluded in POM files aren't pulled in by the Gradle plugin
-   [#&#8203;38721](https://github.com/quarkusio/quarkus/issues/38721) - Java 21: `@VirtualThreadUnit` produces very slow tests
-   [#&#8203;38763](https://github.com/quarkusio/quarkus/issues/38763) - Enable an injection of the OIDC code flow access token verificaton material
-   [#&#8203;38767](https://github.com/quarkusio/quarkus/pull/38767) - Fail early if OIDC client password grant is misconfigured
-   [#&#8203;38771](https://github.com/quarkusio/quarkus/pull/38771) - Adds an implementation note about `@VirtualThreadUnit` limitations
-   [#&#8203;38775](https://github.com/quarkusio/quarkus/pull/38775) - Use the right MongoDB ClientSession interface
-   [#&#8203;38776](https://github.com/quarkusio/quarkus/issues/38776) - OidcRequestFilter with OidcEndpoint applied to all endpoints
-   [#&#8203;38777](https://github.com/quarkusio/quarkus/issues/38777) - OIDC Code flow access token verification goes ahead even if the ID token verification has failed
-   [#&#8203;38779](https://github.com/quarkusio/quarkus/pull/38779) - Fix OidcEndpoint annotation processing
-   [#&#8203;38784](https://github.com/quarkusio/quarkus/pull/38784) - Fix guide URL in RESTEasy Client extension
-   [#&#8203;38785](https://github.com/quarkusio/quarkus/pull/38785) - ArC: fix interception when some methods return void
-   [#&#8203;38798](https://github.com/quarkusio/quarkus/issues/38798) - Using custom header in REST client together with `@NotBody` annotated argument results in warning from EndpointIndexer
-   [#&#8203;38800](https://github.com/quarkusio/quarkus/pull/38800) - Don't warn about `@NotBody` use in `@GET` methods in REST Client
-   [#&#8203;38802](https://github.com/quarkusio/quarkus/issues/38802) - Multipart form data is interpreted as a file although it's not a file
-   [#&#8203;38803](https://github.com/quarkusio/quarkus/issues/38803) - OIDC server is erroneously shown as not available
-   [#&#8203;38810](https://github.com/quarkusio/quarkus/pull/38810) - Expand types which are considered text in multipart handling
-   [#&#8203;38815](https://github.com/quarkusio/quarkus/issues/38815) - Support security identity propagation in VT
-   [#&#8203;38816](https://github.com/quarkusio/quarkus/pull/38816) - Propagate Vert.x context on all ExecutorService methods for VirtualThreadExecutor
-   [#&#8203;38817](https://github.com/quarkusio/quarkus/issues/38817) - Mocking Singleton does not work even when using `@MockitoConfig`(convertScopes = true) - Bean produced from factory method
-   [#&#8203;38818](https://github.com/quarkusio/quarkus/pull/38818) - Allow `RunAndCheckMojoTestBase` subclasses to override how much memory extension tests are allowed
-   [#&#8203;38819](https://github.com/quarkusio/quarkus/pull/38819) - Add response text to the OIDC bootstrap log errors
-   [#&#8203;38821](https://github.com/quarkusio/quarkus/pull/38821) - Configure SISU bean filtering for the bootstrap Maven resolver
-   [#&#8203;38824](https://github.com/quarkusio/quarkus/issues/38824) - Memory leak when using FT Fallback with dependent beans
-   [#&#8203;38833](https://github.com/quarkusio/quarkus/issues/38833) - Keycloak Admin Client Reactive error id: [`9009f9b`](https://github.com/quarkusio/quarkus/commit/9009f9b4)-1d58-4011-9ff2-49b87bb59ddd-1: java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because "authHeader" is null
-   [#&#8203;38836](https://github.com/quarkusio/quarkus/pull/38836) - Fix Keycloak Admin Client Reactive Jackson reader provider priority so that the client can work when the JSONB REST client extension is present
-   [#&#8203;38837](https://github.com/quarkusio/quarkus/issues/38837) - Quarkus create new project fails when -DnoCode is used and artifactId is not set properly
-   [#&#8203;38843](https://github.com/quarkusio/quarkus/pull/38843) - Check the code flow access token after ID token
-   [#&#8203;38844](https://github.com/quarkusio/quarkus/pull/38844) - Fix copy/paste typo
-   [#&#8203;38849](https://github.com/quarkusio/quarkus/pull/38849) - Ensure that generated project GAV is always set
-   [#&#8203;38851](https://github.com/quarkusio/quarkus/issues/38851) - Kafka integration tests fail with latest Mandrel/GraalVM 24.1-dev builds
-   [#&#8203;38853](https://github.com/quarkusio/quarkus/pull/38853) - \[3.7] Perform security checks on inherited endpoints before payload deserialization in the RESTEasy Reactive
-   [#&#8203;38855](https://github.com/quarkusio/quarkus/pull/38855) - Make registration of OAuthBearerValidatorCallbackHandler conditional
-   [#&#8203;38858](https://github.com/quarkusio/quarkus/pull/38858) - Testing: fix `@MockitoConfig`(convertScopes=true) with auto-producers
-   [#&#8203;38859](https://github.com/quarkusio/quarkus/pull/38859) - Fix warning when launching dev mode specifying quarkus-maven-plugin GAV on the command line
-   [#&#8203;38865](https://github.com/quarkusio/quarkus/pull/38865) - Update commons-compress version to mitigate CVE-2024-25710
-   [#&#8203;38866](https://github.com/quarkusio/quarkus/issues/38866) - Sporadic error in custom readiness check using `keycloak-admin-client`: `IllegalStateException: Client is closed`
-   [#&#8203;38868](https://github.com/quarkusio/quarkus/pull/38868) - Add config flag to disable jacoco
-   [#&#8203;38882](https://github.com/quarkusio/quarkus/pull/38882) - Quartz - prevent memory leak when Job instance is a `@Dependent` bean
-   [#&#8203;38886](https://github.com/quarkusio/quarkus/pull/38886) - Ignore `ValidationSchema` that results in registering all models
-   [#&#8203;38888](https://github.com/quarkusio/quarkus/pull/38888) - SmallRye Health: terminate request context properly
-   [#&#8203;38889](https://github.com/quarkusio/quarkus/issues/38889) - Kafka reactive messaging extension incompatible with Micrometer Prometheus extension for Quarkus 3.7.\*
-   [#&#8203;38890](https://github.com/quarkusio/quarkus/pull/38890) - Log resolved OIDC tenant id and how the bearer token is found
-   [#&#8203;38894](https://github.com/quarkusio/quarkus/pull/38894) - Disable messaging observation by default for backwards compatibility
-   [#&#8203;38897](https://github.com/quarkusio/quarkus/pull/38897) - Attempt to fix flaky DependentBeanJobTest

### [`v3.7.3`](https://github.com/quarkusio/quarkus/releases/tag/3.7.3)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.7.2...3.7.3)

##### Complete changelog

-   [#&#8203;36341](https://github.com/quarkusio/quarkus/issues/36341) - The API method KafkaStreams#cleanUp() is not applicable when use `@Produces` to build the topology
-   [#&#8203;37091](https://github.com/quarkusio/quarkus/pull/37091) - Fix VertxGrpcExporter reponse status handling
-   [#&#8203;37911](https://github.com/quarkusio/quarkus/pull/37911) - Store since JavaDoc tag in the configuration metadata, so that Quarkiverse projects can render it in their documentation if they like
-   [#&#8203;38055](https://github.com/quarkusio/quarkus/issues/38055) - Make annotation app.quarkus.io/vcs-uri optional in Kubernetes extension
-   [#&#8203;38079](https://github.com/quarkusio/quarkus/pull/38079) - Make OidcTestSecurityIdentityAugmentor faster by making privateKey's generation final and static
-   [#&#8203;38196](https://github.com/quarkusio/quarkus/pull/38196) - Use Vert.x pool with Jackson
-   [#&#8203;38477](https://github.com/quarkusio/quarkus/pull/38477) - Add disabled workflow to deploy snapshots in Quarkiverse extensions
-   [#&#8203;38489](https://github.com/quarkusio/quarkus/issues/38489) - OIDC authentication.extra-params not added to dev-services auth request
-   [#&#8203;38602](https://github.com/quarkusio/quarkus/issues/38602) - QuarkusComponentTest: `@TestConfigProperties` not applicable to method (override multiple config properties)
-   [#&#8203;38607](https://github.com/quarkusio/quarkus/pull/38607) - Gradle: fix IllegalStateException when resolving project deps
-   [#&#8203;38613](https://github.com/quarkusio/quarkus/issues/38613) - RabbitMQ Health Checks cannot be disabled from 3.7+
-   [#&#8203;38615](https://github.com/quarkusio/quarkus/pull/38615) - Updates to Infinispan 14.0.24.Final
-   [#&#8203;38619](https://github.com/quarkusio/quarkus/pull/38619) - Pass extra authentication params in the OIDC DevUI code flow redirect URL
-   [#&#8203;38626](https://github.com/quarkusio/quarkus/pull/38626) - Bump org.junit.jupiter:junit-jupiter from 5.10.1 to 5.10.2
-   [#&#8203;38650](https://github.com/quarkusio/quarkus/issues/38650) - UI doesn't work correct with umlauts
-   [#&#8203;38653](https://github.com/quarkusio/quarkus/pull/38653) - Enforce Dev UI charset to UTF-8
-   [#&#8203;38655](https://github.com/quarkusio/quarkus/pull/38655) - Allow for multiple TestConfigProperty annotations on methods
-   [#&#8203;38656](https://github.com/quarkusio/quarkus/pull/38656) - Upgrade the Mutiny Vert.x bindings to 3.9.0
-   [#&#8203;38658](https://github.com/quarkusio/quarkus/issues/38658) - Configure a REST Client ClientLogger vía CDI
-   [#&#8203;38662](https://github.com/quarkusio/quarkus/pull/38662) - Bump io.smallrye.config:smallrye-config-source-yaml from 3.5.2 to 3.5.4 in /devtools/gradle
-   [#&#8203;38663](https://github.com/quarkusio/quarkus/issues/38663) - ContainerRequestContext.getUriInfo().getMatchedURIs() IndexOutOfBoundsException
-   [#&#8203;38664](https://github.com/quarkusio/quarkus/pull/38664) - Bump Smallrye RM from 4.16.0 to 4.16.1
-   [#&#8203;38670](https://github.com/quarkusio/quarkus/pull/38670) - Make ClientLogger beans unremovable
-   [#&#8203;38671](https://github.com/quarkusio/quarkus/pull/38671) - Redis Client: improve documentation for sentinel and cluster
-   [#&#8203;38672](https://github.com/quarkusio/quarkus/pull/38672) - Remove WATCH Command in absence of Optimistic Locking
-   [#&#8203;38673](https://github.com/quarkusio/quarkus/pull/38673) - Fix OidcRequestFiler typo in security docs
-   [#&#8203;38674](https://github.com/quarkusio/quarkus/pull/38674) - Improve flaky test
-   [#&#8203;38675](https://github.com/quarkusio/quarkus/pull/38675) - Correct example generated yaml in extension metadata docs
-   [#&#8203;38676](https://github.com/quarkusio/quarkus/issues/38676) - OpenAPI does not fill roles in SecurityScheme in schema
-   [#&#8203;38680](https://github.com/quarkusio/quarkus/pull/38680) - Log how Keycloak devservice maps resources
-   [#&#8203;38681](https://github.com/quarkusio/quarkus/pull/38681) - Upgrade to Hibernate ORM 6.4.4.Final / bytebuddy 1.14.11
-   [#&#8203;38686](https://github.com/quarkusio/quarkus/pull/38686) - Make GraphQL Metrics End when Exceptional
-   [#&#8203;38692](https://github.com/quarkusio/quarkus/pull/38692) - Bump com.gradle:gradle-enterprise-maven-extension from 1.20 to 1.20.1
-   [#&#8203;38693](https://github.com/quarkusio/quarkus/pull/38693) - Bump commons-codec:commons-codec from 1.16.0 to 1.16.1
-   [#&#8203;38694](https://github.com/quarkusio/quarkus/pull/38694) - OpenAPI: remove check that avoids running auto-security at build
-   [#&#8203;38703](https://github.com/quarkusio/quarkus/issues/38703) - RESTEasy Reactive Multipart struggles with non-file binary uploads
-   [#&#8203;38705](https://github.com/quarkusio/quarkus/pull/38705) - Kafka Streams fire event after created and before scheduling the start
-   [#&#8203;38706](https://github.com/quarkusio/quarkus/issues/38706) - Elasticsearch container reuse creates a new container on each run
-   [#&#8203;38709](https://github.com/quarkusio/quarkus/pull/38709) - Don't provide empty paths when using a root prefix
-   [#&#8203;38710](https://github.com/quarkusio/quarkus/pull/38710) - Avoid Vert.x GraphQL deprecation warning
-   [#&#8203;38712](https://github.com/quarkusio/quarkus/pull/38712) - Bump Smallrye RM from 4.16.1 to 4.16.2
-   [#&#8203;38713](https://github.com/quarkusio/quarkus/pull/38713) - Only configure shared network for Elasticsearch/OpenSearch containers where necessary
-   [#&#8203;38714](https://github.com/quarkusio/quarkus/pull/38714) - Don't assume that multipart part without filename is always text
-   [#&#8203;38728](https://github.com/quarkusio/quarkus/pull/38728) - Encode Kafka messages with UTF8
-   [#&#8203;38730](https://github.com/quarkusio/quarkus/issues/38730) - Accept-Header in hibernate validator's ResteasyReactiveLocaleResolver is resolved case-sensitive
-   [#&#8203;38732](https://github.com/quarkusio/quarkus/issues/38732) - Quarkus should still allow to create project with Java 11 (for older streams and other platforms)
-   [#&#8203;38733](https://github.com/quarkusio/quarkus/pull/38733) - Allow Java 11 as LTS for older streams and other platforms
-   [#&#8203;38738](https://github.com/quarkusio/quarkus/pull/38738) - Make accept header check in validation case insensitive
-   [#&#8203;38748](https://github.com/quarkusio/quarkus/pull/38748) - Sanitize app.dekorate.io/vcs-url kubernetes annotation
-   [#&#8203;38755](https://github.com/quarkusio/quarkus/pull/38755) - Log when a RestEasy Reactive client close method is called
-   [#&#8203;38756](https://github.com/quarkusio/quarkus/pull/38756) - Bump Keycloak version to 23.0.6
-   [#&#8203;38760](https://github.com/quarkusio/quarkus/pull/38760) - Set COMPILE_ONLY flag on relevant dependencies that appear on DEPLOYMENT_CP and RUNTIME_CP

### [`v3.7.2`](https://github.com/quarkusio/quarkus/releases/tag/3.7.2)

[Compare Source](https://github.com/quarkusio/quarkus/compare/3.7.1...3.7.2)

##### Complete changelog

-   [#&#8203;37807](https://github.com/quarkusio/quarkus/issues/37807) - SSL requests hang when returning a CompletableFuture
-   [#&#8203;38101](https://github.com/quarkusio/quarkus/issues/38101) - smallrye-openapi property `oidc-open-id-connect-url` might not be fixed at build time
-   [#&#8203;38231](https://github.com/quarkusio/quarkus/pull/38231) - OpenAPI: Always run OpenIDConnectSecurityFilter at runtime
-   [#&#8203;38310](https://github.com/quarkusio/quarkus/pull/38310) - Add note about the two quarkus-extension files
-   [#&#8203;38394](https://github.com/quarkusio/quarkus/issues/38394) - quarkus-cache: "keyGenerator" destroyed, even if it is annotated with "Singleton"
-   [#&#8203;38397](https://github.com/quarkusio/quarkus/pull/38397) - Use actions/setup-java GPG key feature
-   [#&#8203;38411](https://github.com/quarkusio/quarkus/pull/38411) - Cache: only dependent CacheKeyGenerator beans are destroyed after use
-   [#&#8203;38422](https://github.com/quarkusio/quarkus/issues/38422) - nested configurations in extension: sub-property is seen as nested entity.
-   [#&#8203;38431](https://github.com/quarkusio/quarkus/issues/38431) - `quarkus.oidc-token-propagation-reactive.enabled-during-authentication` does not work correctly in the code flow
-   [#&#8203;38442](https://github.com/quarkusio/quarkus/pull/38442) - Make sure the code flow access token is propagated during the authentication
-   [#&#8203;38444](https://github.com/quarkusio/quarkus/pull/38444) - Fix request hanging condition
-   [#&#8203;38451](https://github.com/quarkusio/quarkus/issues/38451) - Remove workaround for HHH-17683 in Panache
-   [#&#8203;38479](https://github.com/quarkusio/quarkus/issues/38479) - Stricter and false positive env variables validation after upgrade to 3.7.0
-   [#&#8203;38483](https://github.com/quarkusio/quarkus/pull/38483) - Add a tool to check cross references
-   [#&#8203;38488](https://github.com/quarkusio/quarkus/pull/38488) - Update to Vert.x 4.5.2
-   [#&#8203;38495](https://github.com/quarkusio/quarkus/pull/38495) - Add org.graalvm.regex:regex to runnerParentFirstArtifacts
-   [#&#8203;38499](https://github.com/quarkusio/quarkus/issues/38499) - Alpn property not work in rest client reactive
-   [#&#8203;38500](https://github.com/quarkusio/quarkus/pull/38500) - Make quarkus.rest-client.alpn work in programmatically created client
-   [#&#8203;38506](https://github.com/quarkusio/quarkus/issues/38506) - lombok warning when building with 3.7.1
-   [#&#8203;38514](https://github.com/quarkusio/quarkus/issues/38514) - Alpn property not work for single rest client reactive
-   [#&#8203;38516](https://github.com/quarkusio/quarkus/pull/38516) - Add missing alpn config key handling from named config
-   [#&#8203;38521](https://github.com/quarkusio/quarkus/issues/38521) - Panache sorting no longer works for embedded fields in Quarkus 3.7.1
-   [#&#8203;38525](https://github.com/quarkusio/quarkus/pull/38525) - Fix typo in RedisClientConfig JavaDoc
-   [#&#8203;38527](https://github.com/quarkusio/quarkus/pull/38527) - Revert "Escape column names with backticks in order by clause of hql query"
-   [#&#8203;38543](https://github.com/quarkusio/quarkus/issues/38543) - LinksProcessor ID field error for native class HalCollectionWrapper
-   [#&#8203;38545](https://github.com/quarkusio/quarkus/issues/38545) - Enhance Adding extension section in cli-tooling documentation page
-   [#&#8203;38546](https://github.com/quarkusio/quarkus/pull/38546) - Add globbing pattern to cli-tooling.adoc
-   [#&#8203;38548](https://github.com/quarkusio/quarkus/pull/38548…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants