Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump kubernetes-client-bom from 6.8.1 to 6.9.2 #36312

Merged
merged 5 commits into from
Nov 14, 2023

Conversation

manusa
Copy link
Contributor

@manusa manusa commented Oct 5, 2023

@gsmet
Copy link
Member

gsmet commented Oct 5, 2023

@metacosm let us know if it's all good for you.

@quarkus-bot

This comment has been minimized.

@manusa
Copy link
Contributor Author

manusa commented Oct 6, 2023

...quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner.jar
2023-10-05T18:46:17.4004088Z ========================================================================================================================
2023-10-05T18:46:17.4027720Z GraalVM Native Image: Generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' (executable)...
2023-10-05T18:46:17.4033873Z ========================================================================================================================
2023-10-05T18:46:17.4039943Z For detailed information and explanations on the build output, visit:
2023-10-05T18:46:17.4046045Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
2023-10-05T18:46:17.4052182Z ------------------------------------------------------------------------------------------------------------------------
2023-10-05T18:46:23.5542193Z [1/8] Initializing...                                                                                   (19.2s @ 0.31GB)
2023-10-05T18:46:23.5547264Z  Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
2023-10-05T18:46:23.5548189Z  Graal compiler: optimization level: 2, target machine: x86-64-v3
2023-10-05T18:46:23.5548800Z  C compiler: gcc (redhat, x86_64, 8.5.0)
2023-10-05T18:46:23.5568395Z  Garbage collector: Serial GC (max heap size: 80% of RAM)
2023-10-05T18:46:23.5579655Z  3 user-specific feature(s)
2023-10-05T18:46:23.5587928Z  - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
2023-10-05T18:46:23.5589002Z  - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
2023-10-05T18:46:23.5589806Z  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
2023-10-05T18:46:26.6780858Z os.name
2023-10-05T18:58:51.5995309Z Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-10-05T18:58:52.3209333Z The Native Image build process ran out of memory. (The maximum heap size of the process was set with '-Xmx5g'.)
2023-10-05T18:58:52.3214684Z Please make sure your build system has more memory available or increase the maximum heap size using the '-Xmx' option.
... quarkus-integration-test-openshift-client-999-SNAPSHOT-runner.jar
2023-10-05T19:05:44.4828590Z ========================================================================================================================
2023-10-05T19:05:44.4848101Z GraalVM Native Image: Generating 'quarkus-integration-test-openshift-client-999-SNAPSHOT-runner' (executable)...
2023-10-05T19:05:44.4849244Z ========================================================================================================================
2023-10-05T19:05:44.4852410Z For detailed information and explanations on the build output, visit:
2023-10-05T19:05:44.4855091Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
2023-10-05T19:05:44.4857508Z ------------------------------------------------------------------------------------------------------------------------
2023-10-05T19:05:49.3010847Z [1/8] Initializing...                                                                                   (15.7s @ 0.37GB)
2023-10-05T19:05:49.3024948Z  Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
2023-10-05T19:05:49.3035161Z  Graal compiler: optimization level: 2, target machine: x86-64-v3
2023-10-05T19:05:49.3042377Z  C compiler: gcc (redhat, x86_64, 8.5.0)
2023-10-05T19:05:49.3055473Z  Garbage collector: Serial GC (max heap size: 80% of RAM)
2023-10-05T19:05:49.3074476Z  3 user-specific feature(s)
2023-10-05T19:05:49.3085265Z  - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
2023-10-05T19:05:49.3090867Z  - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
2023-10-05T19:05:49.3094678Z  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
2023-10-05T19:25:06.6318479Z Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded

@manusa
Copy link
Contributor Author

manusa commented Oct 6, 2023

Local build and test passed for kubernetes-client

This is the GraalVM output:

========================================================================================================================
GraalVM Native Image: Generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (6.6s @ 0.39GB)
 Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s)
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
os.name
[2/8] Performing analysis...  [******]                                                                 (330.9s @ 7.34GB)
  34,129 (94.89%) of 35,968 types reachable
  72,469 (84.26%) of 86,007 fields reachable
 335,426 (86.29%) of 388,698 methods reachable
  24,160 types, 51,546 fields, and 214,821 methods registered for reflection
      63 types,    68 fields, and    55 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (11.3s @ 7.27GB)
[4/8] Parsing methods...      [***]                                                                     (11.8s @ 8.09GB)
[5/8] Inlining methods...     [***]                                                                      (5.8s @ 7.79GB)
[6/8] Compiling methods...    [********]                                                                (71.8s @ 7.16GB)
[7/8] Layouting methods...    [****]                                                                    (16.2s @ 7.52GB)
[8/8] Creating image...       [*****]                                                                   (21.9s @ 6.98GB)
 126.30MB (50.46%) for code area:   300,487 compilation units
 123.64MB (49.40%) for image heap:  993,581 objects and 12 resources
 346.79kB ( 0.14%) for other data
 250.28MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  18.33MB io.fabric8.openshift-model-operator-6.9.0.jar       25.77MB byte[] for code metadata
  15.20MB svm.jar (Native Image)                              20.08MB java.lang.Class
  14.33MB io.fabric8.openshift-model-6.9.0.jar                19.13MB byte[] for java.lang.String
  12.81MB java.base                                           16.95MB byte[] for reflection metadata
   9.06MB io.fabric8.kubernetes-model-core-6.9.0.jar           9.78MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.73MB io.fabric8.openshift-model-monitoring-6.9.0.jar      8.82MB java.lang.String
   4.71MB io.fabric8.openshift-model-hive-6.9.0.jar            6.89MB byte[] for general heap data
   2.91MB i.f.k.9.0.jar                                        2.86MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.72MB io.fabric8.openshift-model-operatorhub-6.9.0.jar     2.32MB java.lang.Object[]
   2.44MB io.fabric8.kubernetes-model-flowcontrol-6.9.0.jar    1.83MB java.lang.String[]
  36.76MB for 137 more packages                                8.07MB for 3549 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                      35.6s (7.4% of total time) in 143 GCs | Peak RSS: 13.19GB | CPU load: 11.22
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner (executable)
 /project/quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' in 7m 59s.

This release of the client includes new Kubernetes v1.28 model types for the kubernetes-client and also OpenShift v4.13 model types. The increase in the number of models might be the cause for the surge in the memory usage.

@gsmet
Copy link
Member

gsmet commented Oct 6, 2023

Unfortunately, I'm not sure we can go much higher than 5g on GitHub Actions without having things failing.

Maybe try to set -Dquarkus.native.native-image-xmx=6g here: https://github.com/quarkusio/quarkus/blob/main/.github/workflows/ci-actions-incremental.yml#L46C65-L46C101 in your PRand see how it goes.

@manusa manusa force-pushed the deps/kubernetes-client branch from 3cf07f8 to dc637ed Compare October 6, 2023 09:25
@quarkus-bot quarkus-bot bot added the area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure label Oct 6, 2023
@quarkus-bot

This comment has been minimized.

@metacosm
Copy link
Contributor

metacosm commented Oct 9, 2023

@metacosm let us know if it's all good for you.

No, it's not good for us. Applications don't compile anymore with errors similar to:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project sample-webpage-operator: Compilation failure: Compilation failure: 
[ERROR] /Volumes/CaseSensitiveDev/java-operator-sdk/sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageStandaloneDependentsReconciler.java: accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.BaseFluent cannot implement accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.Visitable
[ERROR]   return type capture#1 of ? is not compatible with capture#2 of ?
[ERROR] /Volumes/CaseSensitiveDev/java-operator-sdk/sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageDependentsWorkflowReconciler.java: accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.BaseFluent cannot implement accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.Visitable
[ERROR]   return type capture#3 of ? is not compatible with capture#4 of ?

Note that this occurs in code that doesn't use this method so there's nothing that can be done on our side as the error occurs outside of our code. The same code works fine with 6.8.1.

@gsmet gsmet added the triage/on-ice Frozen until external concerns are resolved label Oct 16, 2023
@gsmet
Copy link
Member

gsmet commented Oct 16, 2023

@manusa see comment above ^

@manusa
Copy link
Contributor Author

manusa commented Oct 16, 2023

Yes, sorry for not commenting here.

There's some work being done, I created fabric8io/kubernetes-client#5522 to keep track of it.

I'm not sure if @metacosm has been able to try this weekend's snapshots to see if there's any difference.

I tried to create a simple reproducer in the main Fabric8 repository, but was unable to.

@iocanel was doing some research on the subject, but I'm not sure if he found out anything else.

@manusa manusa changed the title Bump kubernetes-client-bom from 6.8.1 to 6.9.0 Bump kubernetes-client-bom from 6.8.1 to 6.9.1 Oct 26, 2023
@manusa manusa force-pushed the deps/kubernetes-client branch from dc637ed to da8eb4e Compare October 26, 2023 16:36
@manusa
Copy link
Contributor Author

manusa commented Oct 26, 2023

Hi @metacosm

I've updated the PR to bump Kubernetes Client to 6.9.1 which contains the fix for the type inference issue. Please check if it solves your issue.

@quarkus-bot

This comment has been minimized.

@manusa
Copy link
Contributor Author

manusa commented Oct 27, 2023

2023-10-26T20:48:17.7071483Z 	[error]: Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.lang.NoSuchMethodError: 'io.fabric8.kubernetes.api.builder.Fluent io.fabric8.kubernetes.api.model.KubernetesListBuilder.accept(io.fabric8.kubernetes.api.builder.Visitor[])'
2023-10-26T20:48:17.7074491Z 	at io.dekorate.ResourceRegistry.lambda$generate$2(ResourceRegistry.java:181)
2023-10-26T20:48:17.7075473Z 	at java.base/java.util.HashMap.forEach(HashMap.java:1421)
2023-10-26T20:48:17.7077453Z 	at io.dekorate.ResourceRegistry.generate(ResourceRegistry.java:173)
2023-10-26T20:48:17.7078342Z 	at io.dekorate.Session.generate(Session.java:293)
2023-10-26T20:48:17.7079051Z 	at io.dekorate.Session.close(Session.java:256)
2023-10-26T20:48:17.7080150Z 	at io.quarkus.kubernetes.deployment.KubernetesProcessor.lambda$build$2(KubernetesProcessor.java:194)
2023-10-26T20:48:17.7081376Z 	at java.base/java.util.Optional.ifPresent(Optional.java:178)
2023-10-26T20:48:17.7082524Z 	at io.quarkus.kubernetes.deployment.KubernetesProcessor.build(KubernetesProcessor.java:142)
2023-10-26T20:48:17.7088949Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-10-26T20:48:17.7090423Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-10-26T20:48:17.7094883Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-10-26T20:48:17.7159281Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-10-26T20:48:17.7160219Z 	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
2023-10-26T20:48:17.7161481Z 	at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
2023-10-26T20:48:17.7162701Z 	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
2023-10-26T20:48:17.7163765Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
2023-10-26T20:48:17.7165540Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
2023-10-26T20:48:17.7166656Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2023-10-26T20:48:17.7167370Z 	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

It seems we need to bump Dekorate's dependency on Sundrio too

@manusa manusa force-pushed the deps/kubernetes-client branch from da8eb4e to d4ac8cb Compare November 2, 2023 14:26
@manusa manusa changed the title Bump kubernetes-client-bom from 6.8.1 to 6.9.1 Bump kubernetes-client-bom from 6.8.1 to 6.9.2 Nov 2, 2023

This comment has been minimized.

@metacosm
Copy link
Contributor

metacosm commented Nov 6, 2023

Update looks OK on our side.
Spoke too soon: things look OK on the JOSDK side but fail to compile in QOSDK with:
[ERROR] /Volumes/CaseSensitiveDev/quarkus-operator-sdk/core/deployment/src/main/java/io/quarkiverse/operatorsdk/deployment/CRDGeneration.java:[161,59] incompatible types: java.lang.Class<capture#1 of ? extends io.fabric8.kubernetes.client.CustomResource> cannot be converted to java.lang.Class>

Looking into it, seems like type inference is stricter now.

@metacosm
Copy link
Contributor

metacosm commented Nov 6, 2023

Moving past that issue, I get other errors similar to:
Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.lang.NoSuchMethodError: 'io.fabric8.kubernetes.api.builder.Fluent io.fabric8.kubernetes.api.model.KubernetesListBuilder.accept(io.fabric8.kubernetes.api.builder.Visitor[])'

@metacosm
Copy link
Contributor

metacosm commented Nov 6, 2023

The error I'm seeing is consistent with CI here. I guess we need a dekorate upgrade as well.

@manusa manusa force-pushed the deps/kubernetes-client branch from d4ac8cb to 55ebb24 Compare November 6, 2023 10:33
@quarkus-bot quarkus-bot bot added the area/dependencies Pull requests that update a dependency file label Nov 6, 2023
@manusa
Copy link
Contributor Author

manusa commented Nov 6, 2023

I updated the PR to include Dekorate's updated version that Ioannis just released.

@manusa manusa force-pushed the deps/kubernetes-client branch from 3c45f8d to 10b2dfc Compare November 10, 2023 14:36
@manusa
Copy link
Contributor Author

manusa commented Nov 10, 2023

I've got something working, but I'll need some feedback on this.

The main issue with the GraalVM out of memory error is regarding the presence of a really big model module (io.fabric8:openshift-model-operator).

This model will only be required by a fraction of users and for sure is not part of the integration test suite. An easy way to overcome the out of memory problem is to exclude this module in the pom.xml. However, if we exclude it we hit #25526 and the GraalVM build is errored.

My proposed solution is to add a config property: quarkus.kubernetes-client.allow-incomplete-model that when set to true, disables the addition of the --link-at-build-time flag. You can see this in the last commit feat(kubernetes-client): allow incomplete model classpath.

I guess this needs a wider consensus though. Or maybe there's a more suitable solution that I didn't think of. @geoand @gastaldi @gsmet

This comment has been minimized.

@geoand
Copy link
Contributor

geoand commented Nov 13, 2023

My proposed solution is to add a config property: quarkus.kubernetes-client.allow-incomplete-model that when set to true, disables the addition of the --link-at-build-time flag. You can see this in the last commit feat(kubernetes-client): allow incomplete model classpath.

I don't think this is the way to go, as we almost never want extensions to be adding global GraalVM flags.

@manusa
Copy link
Contributor Author

manusa commented Nov 13, 2023

I don't think this is the way to go, as we almost never want extensions to be adding global GraalVM flags.

Well, in this case is not adding, but providing a configuration option to prevent Quarkus from adding a flag automatically.

I'd rather see #25526 fixed instead too.

@geoand
Copy link
Contributor

geoand commented Nov 13, 2023

Well, in this case is not adding, but providing a configuration option to prevent Quarkus from adding a flag automatically.

My point is that the an extension is making changes to GraalVMs's globak configuration

@manusa
Copy link
Contributor Author

manusa commented Nov 13, 2023

My point is that the an extension is making changes to GraalVMs's globak configuration

https://github.com/quarkusio/quarkus/blob/93bab5ae065274f910a444bc940ca6349379f008/extensions/jdbc/jdbc-oracle/deployment/src/main/java/io/quarkus/jdbc/oracle/deployment/OracleMetadataOverrides.java#L148C1-L151

It's basically the same as this, but with an option to opt-in.

@geoand
Copy link
Contributor

geoand commented Nov 13, 2023

Yeah, that's a very very special case that we try very hard to avoid

@metacosm
Copy link
Contributor

So what is the proposed solution, @geoand? We need an update to the client in Quarkus.

@geoand
Copy link
Contributor

geoand commented Nov 13, 2023

@manusa is working on it after a chat we had

@manusa
Copy link
Contributor Author

manusa commented Nov 13, 2023

So what is the proposed solution, @geoand? We need an update to the client in Quarkus.

I'm currently working on one. Since this only affects GH actions environment, I'm adding some hack extension to the kubernetes/openshift-client integration test modules that prevents the --link-at-build-time from being passed on to GraalVM only for those two test projects.

Prevent Quarkus from adding --link-at-build-time GraalVM flag

Signed-off-by: Marc Nuri <[email protected]>
@manusa manusa force-pushed the deps/kubernetes-client branch from 10b2dfc to 26713db Compare November 13, 2023 11:39
@geoand geoand added triage/waiting-for-ci Ready to merge when CI successfully finishes and removed triage/invalid This doesn't seem right triage/on-ice Frozen until external concerns are resolved area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure labels Nov 13, 2023
Copy link

quarkus-bot bot commented Nov 13, 2023

Failing Jobs - Building 26713db

Status Name Step Failures Logs Raw logs Build scan
✔️ Gradle Tests - JDK 11
Gradle Tests - JDK 11 Windows Build Failures Logs Raw logs
✔️ Maven Tests - JDK 11
Maven Tests - JDK 11 Windows Build Failures Logs Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

📦 integration-tests/gradle

io.quarkus.gradle.CustomManifestArgumentsTest.shouldContainsSpecificManifestProperty line 37 - More details - Source on GitHub

java.lang.AssertionError: 

Expecting actual not to be null

⚙️ Maven Tests - JDK 11 Windows #

📦 integration-tests/maven

io.quarkus.maven.it.DevMojoIT.testConfigFactoryInAppModuleBannedInCodeGen line 80 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

io.quarkus.maven.it.DevMojoIT.testConfigFactoryInAppModuleBannedInCodeGen line 80 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

@geoand geoand merged commit ea40376 into quarkusio:main Nov 14, 2023
49 of 51 checks passed
@quarkus-bot quarkus-bot bot added this to the 3.6 - main milestone Nov 14, 2023
@quarkus-bot quarkus-bot bot removed the triage/waiting-for-ci Ready to merge when CI successfully finishes label Nov 14, 2023
@manusa manusa deleted the deps/kubernetes-client branch November 14, 2023 05:37
@metacosm
Copy link
Contributor

Woot! 💪

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dependencies Pull requests that update a dependency file area/kubernetes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants