Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

emilienbev · 2024-10-11T09:23:42Z

Motivation

Upgrade the extension to the latest Quarkus LTS and latest Java SDK release.

Changes

Bump Quarkus to 3.17.5
Bump Java SDK to 3.7.7
Update netty substitutions. As the official netty extension targets Netty 4.1.111.Final and our SDK uses 4.1.112.Final, some substitutions have been added to the "delete" step of the shell script.
(Specific change affecting substitutions between 4.1.111.Final and 4.1.112.Final)
Update: Upgrade Netty substitutions to 4.1.115.Final, SDK and Quarkus are aligned.
The legacy @ConfigRoot approach for CouchbaseConfig was changed to the new @ConfigMapping one.
Added the compiler arg -AlegacyConfigRoot=true to the deployment module pom.xml, as the netty substitution still uses it.
Updated the way CouchbaseDevService gets the config fields.
Added an optional Readiness health check, with a configurable timeout quarkus.couchbase.health.readiness.timeout.
Added support for emitting SDK metrics via the Couchbase metrics-micrometer package and quarkus-micrometer.

Native-image

The native image failed upon calling Cluster-related code due to MpsArrayQueue in DefaultEventBus, OrphanReporter and ThresholdLoggingTracer.

A substitution was added to replace that queue with a safe alternative MpscAtomicUnpaddedArrayQueue
Multiple classes have been registered for reflection (for serialization).

@configroot

---------- Upgrade the extension to the latest Quarkus LTS and latest Java SDK release. ------- - Bump Quarkus to 3.15.1 - Bump Java SDK to 3.7.3 - Update netty substitutions. As the official netty extension targets Netty 4.1.111.Final and our SDK uses 4.1.112.Final, some substitutions have been added to the "delete" step of the shell script. Specifically, this method signature has changed between the two versions (4.1.112 has 1 more string endpointIdentificationAlgorithm in the arguments) https://github.com/netty/netty/blob/ebe2aa5b7cd36562a20b024d78ecff47a86874b8/handler/src/main/java/io/netty/handler/ssl/JdkSslClientContext.java#L267-L273 - The legacy @configroot approach for CouchbaseConfig was changed to the new @ConfigMapping one. - Added the compiler arg "-AlegacyConfigRoot=true" to the deployment module pom.xml, as the netty substitution still uses it. - Updated the way CouchbaseDevService gets the config fields.

…ault in DefaultEventBus, register classes for reflection.

…dLoggingTracer, register classes for reflection.

…ses for reflection

…rable timeout.

…scovered in the global config. Co-authored-by: David Nault <[email protected]>

...oyment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseDevUiProcessor.java

…v UI card

deployment/pom.xml

deployment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseProcessor.java

programmatix · 2025-01-09T14:43:03Z

deployment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseProcessor.java

+
+    @BuildStep
+    ReflectiveClassBuildItem reflection() {
+        return ReflectiveClassBuildItem.builder(


I'm surprised/confused at many of these things needing reflective access. Like CoreConflictResolutionType - it doesn't seem to be used in any reflective way, anywhere in the code base.

They're not used in the extension if that's what you mean, but these are all used in a normal functioning app with the Java SDK.
Quarkus apps can add annotations i.e. @RegisterForReflection to their own code, but not to third-party libraries like ours so we need to provide it ourselves.

If we don't register them for reflection here, an app built with the extension will fail to serialize/deserialize these classes as it didn't store the reflection info on their fields or methods.

Please forgive my ignorance. Why does a class like CoreConflictResolutionType need to be serialized/deserialized?

For this one it's a field in CoreBucketSettingsJson, which is created in ClassicCoreBucketManager.getAllBuckets()for example. That gives the bucket from the http response as a JsonNode to the ObjectMapper: Mapper.convertValue(node, CoreBucketSettingsJson.class);.

In the native image, (in my understanding) the information about the class and its fields/methods which would usually be retrieved via reflection isn't saved (or maybe stripped out), so the ObjectMapper would not be able to match the classes' field names or types to construct the object.

So we'll need to update this list every time we do Jackson data binding with a new class? :-/

How does this work with Jackson in general? Like, is there a Quarkus extension for Jackson that knows to preserve reflection info for any type used in an objectMapper.convertValue(Object, Class) statement? Maybe we could use a similar technique to mark Mapper.convertValue?

In the shorter term, would it be simple to preserve reflection info for all classes in a package? We'd still need to maintain the list of packages, but perhaps updates would be less frequent.

(I'm not suggesting this issue should block the merge; just trying to identify areas that could be improved in the future.)

So we'll need to update this list every time we do Jackson data binding with a new class?

Yes indeed. This list was built by running FIT, and seeing when an reflection error would happen and which operation it affected.

Like, is there a Quarkus extension for Jackson that knows to preserve reflection info for any type used in an objectMapper.convertValue(Object, Class) statement?

For Jackson data-binding this ReflectiveClassBuildItem() from Quarkus generates a reflection-config.json that is passed to Graal, which allows us to target classes in third party dependencies. If those classes were in a Quarkus app or directly in this project, we could use annotations such as @RegisterForReflection on the class to do so.
Some other reflection APIs are automatically analyzed by Graal and do not need manual configuration (link to docs).

There is a Jackson extension for Quarkus that does extra processing and produces an ObjectMapper synthetic bean like for our Cluster, but we would have to explicitly use it in the SDK and configure it too. I haven't gone down that path as it seemed it wasn't needed for us.

would it be simple to preserve reflection info for all classes in a package?

That's interesting, I guess so? It would increase the size of the native-image, I'm not sure by how much.

Thank you for the explanation @emilienbev . Tbh I continue to be rather horrified by the potential maintenance burden of this extension (I know you've put a lot of work into that already), and the optimise-at-any-maintenance-cost philosophy taken by Graal and its closed-system assumptions.

Can we schedule some time to look at solutions, particularly that suggestion of just including all reflection info? I would really like to get this extension safe and easily maintainable first (plus have it not be such a time sink for you), and look at what-I-suspect-will-be-micro optimisations later, if there is uptake that warrants it.

Happy to help if needed, though I think I will just get in your way.

Yep would love to chat with you on this. I could use your input on both maintaining this extension, and maintaining the FIT performer which does not tolerate change well.

...t/src/main/java/com/couchbase/quarkus/extension/deployment/nettyhandling/NettyProcessor.java

integration-tests/pom.xml

...ests/src/main/java/com/couchbase/quarkus/extension/it/CouchbaseQuarkusExtensionResource.java

runtime/src/main/java/com/couchbase/quarkus/extension/runtime/CouchbaseRecorder.java

runtime/src/main/java/com/couchbase/quarkus/extension/runtime/health/CouchbaseReadyCheck.java

update-netty-substitutions.sh

…e initialization.

...oyment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseDevUiProcessor.java

… the Cluster, added histogram configuration.

...oyment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseDevUiProcessor.java

programmatix · 2025-01-15T13:51:01Z

deployment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseProcessor.java

+
+    @BuildStep
+    ReflectiveClassBuildItem reflection() {
+        return ReflectiveClassBuildItem.builder(


Thank you for the explanation @emilienbev . Tbh I continue to be rather horrified by the potential maintenance burden of this extension (I know you've put a lot of work into that already), and the optimise-at-any-maintenance-cost philosophy taken by Graal and its closed-system assumptions.

Can we schedule some time to look at solutions, particularly that suggestion of just including all reflection info? I would really like to get this extension safe and easily maintainable first (plus have it not be such a time sink for you), and look at what-I-suspect-will-be-micro optimisations later, if there is uptake that warrants it.

Happy to help if needed, though I think I will just get in your way.

runtime/pom.xml

runtime/src/main/java/com/couchbase/quarkus/extension/runtime/CouchbaseRecorder.java

…cs configuration. Removed try/catch in connection string helper for the Dev UI.

emilienbev added 3 commits October 9, 2024 16:54

Update to Java SDK 3.7.4

b33ec8f

Update readme

8baa417

emilienbev requested a review from a team as a code owner October 11, 2024 09:23

emilienbev added 2 commits November 4, 2024 14:37

Update to Quarkus 3.16.1, SDK to 3.7.5-SNAPSHOT (to change), fix segf…

e81b03c

…ault in DefaultEventBus, register classes for reflection.

Fix segfaults caused by MpscArrayQueue in OrphanReporter and Threshol…

752d968

…dLoggingTracer, register classes for reflection.

emilienbev changed the title ~~Ugrade to Quarkus 3.15.1 and Java SDK 3.7.4~~ Ugrade to Quarkus 3.16.1 and Java SDK 3.7.5 Nov 5, 2024

emilienbev and others added 5 commits November 14, 2024 11:22

Adjusted MpscArrayQueue processing with future SDK changes

3c8a654

Added new processing for instances of MpscArrayQueue.

098d580

Merge branch 'quarkiverse:main' into main

eeeebfa

Additional classes registered for reflection.

427f7eb

Register BucketManager-related classes for reflection.

a8ced2d

emilienbev changed the title ~~Ugrade to Quarkus 3.16.1 and Java SDK 3.7.5~~ Ugrade to Quarkus 3.16.3 and Java SDK 3.7.6 Nov 25, 2024

emilienbev added 3 commits December 5, 2024 11:40

Added reactive Readiness smallrye health check, some transaction clas…

d8a8ec8

…ses for reflection

Updated Java SDK to 3.7.6, updated subtitutions and processing

d113262

Ready Health check now specifies Mgmt and KV services, with a configu…

22db759

…rable timeout.

emilienbev changed the title ~~Ugrade to Quarkus 3.16.3 and Java SDK 3.7.6~~ Ugrade to Quarkus 3.17.3 and Java SDK 3.7.6 Dec 6, 2024

emilienbev added 2 commits December 6, 2024 19:51

Re-enable devservices in the integration test module

4ecb7b3

Remove Management service from waitUntilReady in Health Check

1b513ce

emilienbev requested a review from programmatix December 9, 2024 16:20

emilienbev and others added 4 commits December 10, 2024 14:34

Fix health check, which now pings GCCCP connection on each KV node di…

3f64ed9

…scovered in the global config. Co-authored-by: David Nault <[email protected]>

Run formatting

936cd01

Add links to the Dev UI card

6cfaed9

Add Micrometer metrics support via quarkus-micrometer

fce8724

emilienbev changed the title ~~Ugrade to Quarkus 3.17.3 and Java SDK 3.7.6~~ Ugrade to Quarkus 3.17.4 and Java SDK 3.7.6 Dec 13, 2024

emilienbev added 2 commits January 8, 2025 12:55

Added documentation

93f9a6e

Updated README.md

6a16179

dnault reviewed Jan 8, 2025

View reviewed changes

...oyment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseDevUiProcessor.java Outdated Show resolved Hide resolved

Improved/simplified host extraction from connection string for the De…

d40a7d6

…v UI card

programmatix reviewed Jan 9, 2025

View reviewed changes

Updated Netty substitutions, added CoreHttpRequest$Builder for runtim…

1aac1cb

…e initialization.

dnault reviewed Jan 9, 2025

View reviewed changes

...oyment/src/main/java/com/couchbase/quarkus/extension/deployment/CouchbaseDevUiProcessor.java Outdated Show resolved Hide resolved

emilienbev added 2 commits January 10, 2025 18:12

Cleaned up connection string util, simplified configuring metrics for…

5e12446

… the Cluster, added histogram configuration.

Re-add defaults to integration-tests application.properties

a89791f

programmatix reviewed Jan 15, 2025

View reviewed changes

emilienbev added 2 commits January 16, 2025 16:34

Removed unnecessary dependency and cleaned up CouchbaseRecorder metri…

45ce195

…cs configuration. Removed try/catch in connection string helper for the Dev UI.

Upgrade to SDK 3.7.7 and rework Metrics, docs.

b570585

emilienbev changed the title ~~Ugrade to Quarkus 3.17.4 and Java SDK 3.7.6~~ Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 Jan 17, 2025

emilienbev added 2 commits January 17, 2025 16:39

Update docs on configuring metrics, add license header.

5f076d8

Update metrics-micrometer dependency

7b6b3f0

dnault approved these changes Jan 17, 2025

View reviewed changes

emilienbev merged commit 9bfaa05 into quarkiverse:main Jan 17, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

emilienbev commented Oct 11, 2024 •

edited

Loading

programmatix Jan 9, 2025

emilienbev Jan 9, 2025

dnault Jan 9, 2025

emilienbev Jan 10, 2025 •

edited

Loading

dnault Jan 10, 2025

emilienbev Jan 13, 2025 •

edited

Loading

programmatix Jan 15, 2025

emilienbev Jan 15, 2025

programmatix Jan 15, 2025

Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

Conversation

emilienbev commented Oct 11, 2024 • edited Loading

Motivation

Changes

Native-image

programmatix Jan 9, 2025

Choose a reason for hiding this comment

emilienbev Jan 9, 2025

Choose a reason for hiding this comment

dnault Jan 9, 2025

Choose a reason for hiding this comment

emilienbev Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

dnault Jan 10, 2025

Choose a reason for hiding this comment

emilienbev Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

programmatix Jan 15, 2025

Choose a reason for hiding this comment

emilienbev Jan 15, 2025

Choose a reason for hiding this comment

programmatix Jan 15, 2025

Choose a reason for hiding this comment

emilienbev commented Oct 11, 2024 •

edited

Loading

emilienbev Jan 10, 2025 •

edited

Loading

emilienbev Jan 13, 2025 •

edited

Loading