Native builds fail after upgrading to Quarkus 3.5.1, Mandrel 23.1.1.0-Final and Java 21

[mdvcgn]

Describe the bug

Previously successful native builds of an application are not possible anymore after upgrading the project - see "Actual behavior".
The upgrades conducted are as follows:

• Java 17 -> 21
• Quarkus 3.3.0 -> 3.5.1
• Mandrel 23.0.1.2-Final -> 23.1.1.0-Final

Other projects with the same upgrades are not affected and an analysis of the error log leads me to the conclusion that the Maven dependency com.google.cloud.sql:postgres-socket-factory:1.14.1 is the culprit, maybe something with JNR as those come transitively with that.

Expected behavior

Native builds still work as before after upgrading aforementioned versions

Actual behavior

Native build fails with the following log entries:

Build resources:
 - 6.22GB of memory (77.8% of 8.00GB system memory, set via '-Xmx7g')
 - 2 thread(s) (100.0% of 2 available processor(s), determined at start)
[2/8] Performing analysis...  [*****]                                                                  (153.6s @ 1.13GB)
   13,073 reachable types   (86.0% of   15,209 total)
   18,526 reachable fields  (56.8% of   32,597 total)
   64,703 reachable methods (55.5% of  116,494 total)
    4,179 types,   410 fields, and 4,521 methods registered for reflection
       61 types,    59 fields, and    55 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
Error: Detected a started Thread in the image heap. Thread name: jnr.ffi.util.ref.internal.Finalizer. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use 
    '--trace-object-instantiation=java.lang.Thread'
to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
    '--initialize-at-run-time=<culprit>'
to prevent the instantiation of the object.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: Object was reached by
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$Node.waiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.firstWaiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d
  reading field java.lang.ref.ReferenceQueue.notEmpty of constant 
    java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c
  scanning root java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c embedded in 
    jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java)
  parsing method jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java:132) reachable via the parsing context
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:75)
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:45)
    at jnr.ffi.provider.jffi.NativeMemoryManager.allocateDirect(NativeMemoryManager.java:49)
    at jnr.ffi.Struct$Info.allocateMemory(Struct.java:90)
    at jnr.ffi.Struct$Info.getMemory(Struct.java:69)
    at jnr.ffi.Struct$AbstractMember.getMemory(Struct.java:826)
    at jnr.ffi.Struct$UTFString.getStringMemory(Struct.java:2425)
    at jnr.ffi.Struct$UTFString.get(Struct.java:2429)
    at jnr.ffi.Struct$String.toString(Struct.java:2415)
    at java.lang.String.valueOf(String.java:4461)
    at java.security.Provider.putId(Provider.java:857)
    at java.security.Provider.<init>(Provider.java:238)
    at sun.security.provider.Sun.<init>(Sun.java:53)
    at com.oracle.svm.core.code.FactoryMethodHolder.Sun_constructor_992f12f580cf47c3444e9ec7818b9d0a187cd44d(generated:0)
    at static root method.(Unknown Source)
com.oracle.svm.core.util.UserError$UserException: Detected a started Thread in the image heap. Thread name: jnr.ffi.util.ref.internal.Finalizer. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use 
    '--trace-object-instantiation=java.lang.Thread'
to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
    '--initialize-at-run-time=<culprit>'
to prevent the instantiation of the object.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: Object was reached by
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$Node.waiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.firstWaiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d
  reading field java.lang.ref.ReferenceQueue.notEmpty of constant 
    java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c
  scanning root java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c embedded in 
    jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java)
  parsing method jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java:132) reachable via the parsing context
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:75)
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:45)
    at jnr.ffi.provider.jffi.NativeMemoryManager.allocateDirect(NativeMemoryManager.java:49)
    at jnr.ffi.Struct$Info.allocateMemory(Struct.java:90)
    at jnr.ffi.Struct$Info.getMemory(Struct.java:69)
    at jnr.ffi.Struct$AbstractMember.getMemory(Struct.java:826)
    at jnr.ffi.Struct$UTFString.getStringMemory(Struct.java:2425)
    at jnr.ffi.Struct$UTFString.get(Struct.java:2429)
    at jnr.ffi.Struct$String.toString(Struct.java:2415)
    at java.lang.String.valueOf(String.java:4461)
    at java.security.Provider.putId(Provider.java:857)
    at java.security.Provider.<init>(Provider.java:238)
    at sun.security.provider.Sun.<init>(Sun.java:53)
    at com.oracle.svm.core.code.FactoryMethodHolder.Sun_constructor_992f12f580cf47c3444e9ec7818b9d0a187cd44d(generated:0)
    at static root method.(Unknown Source)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:85)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:248)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:814)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:538)
 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:720)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:142)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:97)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Thread name: jnr.ffi.util.ref.internal.Finalizer. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use 
    '--trace-object-instantiation=java.lang.Thread'
to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
    '--initialize-at-run-time=<culprit>'
to prevent the instantiation of the object.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: Object was reached by
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$Node.waiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@16abede8
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.firstWaiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1201d95d
  reading field java.lang.ref.ReferenceQueue.notEmpty of constant 
    java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c
  scanning root java.lang.ref.ReferenceQueue@22d4697c: java.lang.ref.ReferenceQueue@22d4697c embedded in 
    jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java)
  parsing method jnr.ffi.provider.jffi.TransientNativeMemory$Magazine.<init>(TransientNativeMemory.java:132) reachable via the parsing context
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:75)
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:45)
    at jnr.ffi.provider.jffi.NativeMemoryManager.allocateDirect(NativeMemoryManager.java:49)
    at jnr.ffi.Struct$Info.allocateMemory(Struct.java:90)
    at jnr.ffi.Struct$Info.getMemory(Struct.java:69)
    at jnr.ffi.Struct$AbstractMember.getMemory(Struct.java:826)
    at jnr.ffi.Struct$UTFString.getStringMemory(Struct.java:2425)
    at jnr.ffi.Struct$UTFString.get(Struct.java:2429)
    at jnr.ffi.Struct$String.toString(Struct.java:2415)
    at java.lang.String.valueOf(String.java:4461)
    at java.security.Provider.putId(Provider.java:857)
    at java.security.Provider.<init>(Provider.java:238)
    at sun.security.provider.Sun.<init>(Sun.java:53)
    at com.oracle.svm.core.code.FactoryMethodHolder.Sun_constructor_992f12f580cf47c3444e9ec7818b9d0a187cd44d(generated:0)
    at static root method.(Unknown Source)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:126)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:809)
	... 6 more

How to Reproduce?

Reproducer:

1. Generate example Quarkus project - I used Quarkus create
2. Change maven.compiler.release in pom.xml to 21
3. Add dependency com.google.cloud.sql:postgres-socket-factory:1.14.1
4. Add dependency io.quarkus:quarkus-agroal
5. Add dependency io.quarkus:quarkus-jdbc-postgresql
6. Remove tests in src/test/java to make build not fail with Quarkus test startup failure due to missing Docker
7. Start native build with Mandrel

Output of uname -a or ver

Linux runner-... 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed Apr 27 20:34:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "21.0.1" 2023-10-17 LTS / OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS) / OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode, sharing)

Mandrel or GraalVM version (if different from Java)

Mandrel-23.1.1.0-Final

Quarkus version or git rev

3.5.1

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.4

Additional information

No response

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel)

[zakkak]

Running:

quarkus create
cd code-with-zip
quarkus extension add com.google.cloud.sql:postgres-socket-factory:1.14.1
quarkus extension add io.quarkus:quarkus-agroal
quarkus extension add io.quarkus:quarkus-jdbc-postgresql
rm -rf src/test/java
./mvnw package -Dnative -Dquarkus.native.container-build

I can reproduce the issue with a slightly different stack trace though:

Error: Detected a started Thread in the image heap. Thread name: jnr.ffi.util.ref.internal.Finalizer. Threads running in the image generator are no longer running at image runtime. If these objects should not be stored in the image heap, you can use 

    '--trace-object-instantiation=java.lang.Thread'

to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 

    '--initialize-at-run-time=<culprit>'

to prevent the instantiation of the object.
The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: Object was reached by
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$Node.waiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@47a42b6b: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode@47a42b6b
  reading field java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.firstWaiter of constant 
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42376494: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42376494
  reading field java.lang.ref.ReferenceQueue.notEmpty of constant 
    java.lang.ref.ReferenceQueue@78a7a150: java.lang.ref.ReferenceQueue@78a7a150
  reading field jnr.ffi.util.ref.FinalizableReferenceQueue.queue of constant 
    jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791: jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791
  reading field java.lang.ref.Reference.referent of constant 
    java.util.WeakHashMap$Entry@7deb0f9: jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791=true
  indexing into array java.util.WeakHashMap$Entry[]@520dbfe0: [Ljava.util.WeakHashMap$Entry;@520dbfe0
  reading field java.util.WeakHashMap.table of constant 
    java.util.WeakHashMap@51a2e0ea: {jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791=true}
  reading field java.util.Collections$SynchronizedMap.m of constant 
    java.util.Collections$SynchronizedMap@79517b7c: {jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791=true}
  scanning root java.util.Collections$SynchronizedMap@79517b7c: {jnr.ffi.util.ref.FinalizableReferenceQueue@3de58791=true} embedded in 
    jnr.ffi.util.ref.FinalizableReferenceQueue.cleanUpAll(FinalizableReferenceQueue.java:300)
  parsing method jnr.ffi.util.ref.FinalizableReferenceQueue.cleanUpAll(FinalizableReferenceQueue.java:300) reachable via the parsing context
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:72)
    at jnr.ffi.provider.jffi.TransientNativeMemory.allocate(TransientNativeMemory.java:45)
    at jnr.ffi.provider.jffi.NativeMemoryManager.allocateDirect(NativeMemoryManager.java:49)
    at jnr.ffi.Struct$Info.allocateMemory(Struct.java:90)
    at jnr.ffi.Struct$Info.getMemory(Struct.java:69)
    at jnr.ffi.Struct$AbstractMember.getMemory(Struct.java:826)
    at jnr.unixsocket.SockAddrUnix.strlen(SockAddrUnix.java:178)
    at jnr.unixsocket.SockAddrUnix.length(SockAddrUnix.java:155)
    at jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:120)
    at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:139)
    at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:68)
    at com.google.cloud.sql.core.CoreSocketFactory.connect(CoreSocketFactory.java:213)
    at static root method.(Unknown Source)

As shown, the root cause is com.google.cloud.sql.core.CoreSocketFactory.connect.
Adding -Dquarkus.native.additional-build-args=--trace-object-instantiation=java.lang.Thread to the build command gives some more info:

The culprit object has been instantiated by the 'jnr.ffi.provider.jffi.NativeFinalizer$SingletonHolder' class initializer with the following trace:
	at java.lang.Thread.<init>(Thread.java:1477)
	at java.lang.invoke.LambdaForm$DMH/0x0000000801a12800.newInvokeSpecial(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x0000000801a19000.invoke(LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
	at jdk.internal.reflect.DirectConstructorHandleAccessor.invokeImpl(DirectConstructorHandleAccessor.java:90)
	at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at jnr.ffi.util.ref.internal.Finalizer.start(Finalizer.java:138)
	at jnr.ffi.util.ref.internal.Finalizer.startFinalizer(Finalizer.java:88)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x0000000801a12400.invoke(LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00000008000c5c00.invokeExact_MT(LambdaForm$MH)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke(Method.java:580)
	at jnr.ffi.util.ref.FinalizableReferenceQueue.<init>(FinalizableReferenceQueue.java:115)
	at jnr.ffi.provider.jffi.NativeFinalizer.<init>(NativeFinalizer.java:27)
	at jnr.ffi.provider.jffi.NativeFinalizer$SingletonHolder.<clinit>(NativeFinalizer.java:30)

Passing -Dquarkus.native.additional-build-args=--initialize-at-run-time=jnr.ffi.provider.jffi.NativeFinalizer\$SingletonHolder makes the build succeed.

So we probably need to mark this class as runtime initialized in Quarkus, when using com.google.cloud.sql:postgres-socket-factory.

@geoand @gsmet would that require a new extension for com.google.cloud.sql:postgres-socket-factory?

[geoand]

That would be a question for @loicmathieu to address in https://github.com/quarkiverse/quarkus-google-cloud-services

[zakkak]

That would be a question for @loicmathieu to address in https://github.com/quarkiverse/quarkus-google-cloud-services

Is it though? According to this guide it should work without the quarkus-google-cloud-services extension (unless I missed something).

[loicmathieu]

For the moment there is no extension for Google Cloud SQL so this is out of support. We can of course add one inside the Google Cloud Services extension pack.
But what also could be done is to open an issue inside the Google Cloud Java SDK, this should be inside this repository I think: https://github.com/googleapis/google-cloud-java

We only document how to setup Quarkus datasources for Cloud SQL, we can also add this piece of configuration in the guide.

[mdvcgn]

That would be a question for @loicmathieu to address in https://github.com/quarkiverse/quarkus-google-cloud-services

Is it though? According to this guide it should work without the quarkus-google-cloud-services extension (unless I missed something).

Your assumption is true I think, as in our corresponding project we only use the socket factory dependency, not quarkus-google-cloud-services

[mdvcgn]

Passing -Dquarkus.native.additional-build-args=--initialize-at-run-time=jnr.ffi.provider.jffi.NativeFinalizer\$SingletonHolder makes the build succeed.

Thanks for the hint, I can confirm that this indeed fixes the native build.
I deployed the resulting executable within an image to Cloud Run and everything is working as expected ✔

[zakkak]

For the moment there is no extension for Google Cloud SQL so this is out of support. We can of course add one inside the Google Cloud Services extension pack. But what also could be done is to open an issue inside the Google Cloud Java SDK, this should be inside this repository I think: https://github.com/googleapis/google-cloud-java

There already exists an issue for GraalVM native image support in GoogleCloudPlatform/cloud-sql-jdbc-socket-factory#217 (which is the repo containing the class triggering the issue)

We only document how to setup Quarkus datasources for Cloud SQL, we can also add this piece of configuration in the guide.

I will prepare a PR for that.