java.lang.ClassCastException when upgrading from 2.16.0.Final to 2.16.1.Final

[spolti]

Describe the bug

After upgrading to Quarkus 2.16.1 this issue started to happen:

 Caused by: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
 	[error]: Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.lang.ClassCastException: class java.util.HashMap cannot be cast to class java.lang.String (java.util.HashMap and java.lang.String are in module java.base of loader 'bootstrap')
 	at io.dekorate.docker.adapter.DockerBuildConfigAdapter.newBuilder(DockerBuildConfigAdapter.java:58)
 	at io.dekorate.docker.config.DockerBuildConfigGenerator.addPropertyConfiguration(DockerBuildConfigGenerator.java:48)
 	at io.dekorate.Session.lambda$addPropertyConfiguration$1(Session.java:167)
 	at io.dekorate.Session.addConfiguration(Session.java:183)
 	at io.dekorate.Session.addPropertyConfiguration(Session.java:167)
 	at io.quarkus.kubernetes.deployment.KubernetesProcessor.lambda$build$2(KubernetesProcessor.java:164)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at io.quarkus.kubernetes.deployment.KubernetesProcessor.build(KubernetesProcessor.java:141)
 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
 	at io.quarkus.builder.BuildContext.run(BuildContext.java:281)
 	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
 	at java.base/java.lang.Thread.run(Thread.java:829)
 	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

Expected behavior

No issues when upgrading from 2.16.0.Final to 2.16.1.Final

Actual behavior

This seems to be a breaking change that we would not expect to see in a minor release.

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Sgitario (kubernetes), @geoand (kubernetes), @iocanel (kubernetes)

[radtriste]

See also https://github.com/kiegroup/kogito-runtimes/pull/2783

[Sgitario]

Can you share the configuration you're using?

[radtriste]

@Sgitario https://github.com/kiegroup/kogito-examples/tree/main/kogito-quarkus-examples/onboarding-example/onboarding-quarkus

[radtriste]

that would be:

quarkus.http.port=8080
quarkus.infinispan-client.hosts=localhost:11222
quarkus.infinispan-client.use-auth=false

# Kubernetes related properties
%minikube.quarkus.kubernetes.deployment-target=minikube
%kubernetes.quarkus.kubernetes.deployment-target=kubernetes
quarkus.container-image.group=kogito
quarkus.container-image.name=onboarding-service
quarkus.container-image.tag=latest
quarkus.kubernetes.labels.onboarding=process

%knative.quarkus.kubernetes.deployment-target=kubernetes
%knative.quarkus.knative.labels.onboarding=process

[geoand]

Looks like a dekotate issue. @Sgitario we should probably add some kind of test in Quarkus to make sure this never happens again

[Sgitario]

We already have tests to cover this scenario.
I could not reproduce this issue when building the Kogito example locally and using Quarkus 2.16.1.Final (I simply tried mvn clean install). Let me know if I need to use something else to reproduce it.

Also, are you adding some properties when running Maven? I could not find any in the Jenkins logs though :/
Quarkus Kubernetes is automatically mapping some properties into Dekorate properties and maybe here is the issue, but I can't be 100% sure.

[geoand]

We certainly have tests, but something is obviously wrong in this case.
It would be nice to add a test once we know that the actual problem is

[Sgitario]

We certainly have tests, but something is obviously wrong in this case. It would be nice to add a test once we know that the actual problem is

Indeed

[radtriste]

command line:

mvn clean install  -s /home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-runtimes.tests.downstream.kogito-quarkus-examples@tmp/config1980440886206568513tmp -Dmaven.wagon.http.ssl.insecure=true -Dmaven.test.failure.ignore=true -nsu -ntp -fae -e -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 -Dmaven.wagon.http.retryHandler.count=3   dependency:tree  -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B

nothing quarkus specific here

Trying to get quick reproducer

[Sgitario]

Thanks

To give more information, the dekorate version is the same for 2.16.0.Final and 2.16.1.Final.
In Quarkus, as far I see, the only difference in the Kubernetes extension between these releases is #30591, which seems related. Though, I don't see how these changes could break your example.

[radtriste]

ok. cannot reproduce as well ...
so seems an issue on CI ...

[spolti]

@radtriste can this be closed?

[radtriste]

still having an issue here ... but cannot reproduce ...
GHA is green though ...

[radtriste]

Could it be a Filesystem issue on Jenkins ?

[radtriste]

I mean reading the properties ?

[radtriste]

The issue is happening really on 3 apps and not the other. But the container configuration is on almost all examples

[spolti]

is there a chance of something be pulling and older version of Quarkus and, maybe the dekorate dep?

[radtriste]

dekorate-core and docker-annotations are both in version 3.2.0

[radtriste]

So far I tried different hypothesis on why it fails on Jenkins:

1. Hierarchy of the repositories in the filesystem
2. Environment variables already available on Jenkins node
3. Application properties set (but why would it work on GHA ?)

If you want to help, here are some links (Red Hat internal):

• Example used is here
• A failure example, just for one example and with debug enabled, can be found here
• The environment can be found here

[Sgitario]

I took another look into this issue and there is something else that rings a bell to me: why the docker build is triggered? As far as I see, you're only using the container-image-jib extension and this is only enabled when using the "container" profile.

After digging a bit on this, I found out that the docker-annotations module from dekorate is added when using the kubernetes-annotations module (used by the quarkus-kubernetes extension) which seems unexpected to me.

To continue with, by default the docker-annotations module does nothing (it's not even triggered). However, if we add any system property with the prefix docker.XX=YY, this module is going to be enabled (I guess dekorate is listening on any of these properties) and that's why is failing in your CI environment, and not for us in our local machine (we don't provide any property of style docker.XX=YY.

We need to fix this issue in Quarkus or in Dekorate. Also, I'm trying to find a quick workaround, but if we exclude the docker-annotations dependency from quarkus-kubernetes, dekorate will fail with:

[ERROR]         [error]: Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.util.ServiceConfigurationError: io.dekorate.BuildServiceFactory: Provider io.dekorate.docker.buildservice.DockerBuildServiceFactory not found
[ERROR]         at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
[ERROR]         at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)

I'm not sure if you can somehow remove all the properties that starts with docker. using a Maven plugin or move the quarkus-kubernetes extension altogether to another profile.

[Sgitario]

FYI: This issue is going to be fixed by #30977. but this pull request cannot be backported to 2.16 because it has breaking changes.

Therefore, let me know if you managed to address the issue by either moving the quarkus-kubernetes extension to another profile or removing the docker. properties. Otherwise, I can provide a quick fix only for 2.16 (though, there are no plans to release 2.16.2, right @gsmet ?)

[gsmet]

@Sgitario I'm releasing 2.16.2.Final as we speak and there will be more 2.16 micros for sure. So please fix it in 2.16 too if you can.

[Sgitario]

Here you have: #30992
I've manually verified that now the Docker dekorate builder is not triggered when adding any docker.XX=YY property. So, this should address this issue.

[radtriste]

Let me know when it is released or at least merged.
the update of Kogito will be done directly to 2.16.2+ (skipping 2.16.1)

[radtriste]

in fact, @Sgitario I am running directly with your branch to check this was the fix :)

[Sgitario]

in fact, @Sgitario I am running directly with your branch to check this was the fix :)

Great! Let me know if it was ok!

[radtriste]

@Sgitario @gsmet all greem with change from Jose.
Will wait for the next bug fix release :)

[radtriste]

Should be fixed in #30992

[scrocquesel]

Maybe of interest, I have CI env var named DOCKER_REGISTRY_USER and DOCKER_REGISTRY_PASSWORD which seems to be converted to a map for the key docker.registry but docker-annotation expects a string. Removing them fix the build meanwhile.

I don't use docker-annotation in the project and it's indeed a good idea to get rid of it from the ground up, but I wonder what would be the long term fix. I guess the generated adapter in dekorate should also not assume the type of the value is what it expects especially with so common variable name.

[gsmet]

@Sgitario not sure if you have seen this one ^. I think we could close the issue but maybe there is more to it?

[Sgitario]

I think we can close it as it's already fixed in the main branch and in 2.16.

[spolti]

Many thanks all!!

[radtriste]

@gsmet @Sgitario Can we expect 7afcad6 to be in a Quarkus 2.16 release ?

[radtriste]

or is it part of 2.16.4 ? (could not get the information)