Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ugrade to Quarkus 3.17.5 and Java SDK 3.7.7 #123

Merged
merged 29 commits into from
Jan 17, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
500fbdf
Upgrade to Quarkus 3.15.1 and SDK 3.7.3
emilienbev Oct 9, 2024
b33ec8f
Update to Java SDK 3.7.4
emilienbev Oct 10, 2024
8baa417
Update readme
emilienbev Oct 11, 2024
e81b03c
Update to Quarkus 3.16.1, SDK to 3.7.5-SNAPSHOT (to change), fix segf…
emilienbev Nov 4, 2024
752d968
Fix segfaults caused by MpscArrayQueue in OrphanReporter and Threshol…
emilienbev Nov 4, 2024
3c8a654
Adjusted MpscArrayQueue processing with future SDK changes
emilienbev Nov 14, 2024
098d580
Added new processing for instances of MpscArrayQueue.
emilienbev Nov 14, 2024
eeeebfa
Merge branch 'quarkiverse:main' into main
emilienbev Nov 19, 2024
427f7eb
Additional classes registered for reflection.
emilienbev Nov 22, 2024
a8ced2d
Register BucketManager-related classes for reflection.
emilienbev Nov 25, 2024
d8a8ec8
Added reactive Readiness smallrye health check, some transaction clas…
emilienbev Dec 5, 2024
d113262
Updated Java SDK to 3.7.6, updated subtitutions and processing
emilienbev Dec 6, 2024
22db759
Ready Health check now specifies Mgmt and KV services, with a configu…
emilienbev Dec 6, 2024
4ecb7b3
Re-enable devservices in the integration test module
emilienbev Dec 6, 2024
1b513ce
Remove Management service from waitUntilReady in Health Check
emilienbev Dec 9, 2024
3f64ed9
Fix health check, which now pings GCCCP connection on each KV node di…
emilienbev Dec 10, 2024
936cd01
Run formatting
emilienbev Dec 10, 2024
6cfaed9
Add links to the Dev UI card
emilienbev Dec 11, 2024
fce8724
Add Micrometer metrics support via quarkus-micrometer
emilienbev Dec 13, 2024
93f9a6e
Added documentation
emilienbev Jan 8, 2025
6a16179
Updated README.md
emilienbev Jan 8, 2025
d40a7d6
Improved/simplified host extraction from connection string for the De…
emilienbev Jan 9, 2025
1aac1cb
Updated Netty substitutions, added CoreHttpRequest$Builder for runtim…
emilienbev Jan 9, 2025
5e12446
Cleaned up connection string util, simplified configuring metrics for…
emilienbev Jan 10, 2025
a89791f
Re-add defaults to integration-tests application.properties
emilienbev Jan 11, 2025
45ce195
Removed unnecessary dependency and cleaned up CouchbaseRecorder metri…
emilienbev Jan 16, 2025
b570585
Upgrade to SDK 3.7.7 and rework Metrics, docs.
emilienbev Jan 17, 2025
5f076d8
Update docs on configuring metrics, add license header.
emilienbev Jan 17, 2025
7b6b3f0
Update metrics-micrometer dependency
emilienbev Jan 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,15 @@ This extension is currently in beta status. It supports:
- Dependency injecting a Couchbase `Cluster`.
- Configuring the Cluster through `application.properties`. Currently, a minimal set of configuration options is provided.
- A dev service that starts a Couchbase server in a Docker container. With this you can develop your Quarkus app without having to install Couchbase on your machine.
- GraalVM/Mandrel/native-image.
- KV, Query, Transactions, Analytics, Search and Management operations.
- Micrometer metrics using `quarkus-micrometer`
- SmallRye Health checks (Readiness) using `quarkus-smallrye-health`

Please try it out and provide feedback, ideas and bug reports [on Github](https://github.com/quarkiverse/quarkus-couchbase/issues).

## Native Image
> [!IMPORTANT]
> While the extension compiles natively, there are known issues with the produced runner which we are actively working to resolve.
All main Cluster operations have been tested with our internal testing tools, however the extension remains in beta and some features may be missing or not be fully supported.

## Usage
Add it to your project:
Expand Down Expand Up @@ -90,9 +93,13 @@ public class TestCouchbaseResource {

And test http://localhost:8080/couchbase/test.

## Additional Configuration
Please refer to the [docs](https://github.com/quarkiverse/quarkus-couchbase/blob/main/docs/modules/ROOT/pages/configuration.adoc) for additional configuration options.

## Limitations
In this a beta release the configuration options are limited to the three shown above.
The Cluster configuration options are limited cluster credentials, and micrometer metrics emission rate.
This means that a Couchbase cluster configured securely and requiring TLS or a client or server certificate, cannot currently be connected to.
Additional Cluster connections can be created and configured using `Cluster.connect(...)`, however not all code paths have been tested and are therefore not officially supported.

## License
All the files under `nettyhandling` directories, both in the `runtime` and `deployment` modules are
Expand Down
28 changes: 22 additions & 6 deletions deleteMethod.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,35 @@ count_braces() {
local file="$2"

perl -ne '
BEGIN { $count = 0; $found = 0; $start_line = 0; }
if (/\b'"$method"'\s*\(/) { $found = 1; $start_line = $.; }
BEGIN { $count = 0; $found = 0; $in_method = 0; $start_line = 0; }

# Find the method or class name and ignore leading spaces
if (!$in_method && /^\s*\b'"$method"'\s*(\(\s*\))?/) {
$in_method = 1;
$start_line = $.;
}

# Go through each line until we find an opening brace (for multi-line method arguments)
if ($in_method) {
if (/.*\{/) {
$found = 1;
$in_method = 0;
}
}

# Once inside the body, count braces to support nested ones and print start and end when reaching the last
if ($found) {
$count += tr/{/{/;
$count -= tr/}/}/;
if ($count == 0 && $found == 1) {
if ($count == 0) {
print "START:$start_line | END:$.\n";
exit;
}
}
' "$file"
}


delete_method() {
local method="$1"
local file="$2"
Expand All @@ -29,11 +45,11 @@ delete_method() {
end_line=$(echo "$line_info" | cut -d':' -f3)

if [ -n "$start_line" ] && [ -n "$end_line" ]; then
start_line=$((start_line - 1))
start_line=$((start_line - 1)) # start_line -1 to delete the annotation that comes before
sed -i '' "${start_line},${end_line}d" "$file"
echo "Deleted method '$method' from $file (Lines: $start_line - $end_line)"
echo "Deleted method/class '$method' from $file (Lines: $start_line - $end_line)"
else
echo "Method '$method' not found in $file"
echo "Method/Class '$method' not found in $file"
fi
}

Expand Down
7 changes: 7 additions & 0 deletions deployment/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@
<artifactId>quarkus-couchbase</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-health-spi</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5-internal</artifactId>
Expand All @@ -41,6 +45,9 @@
<version>${quarkus.version}</version>
</path>
</annotationProcessorPaths>
<compilerArgs>
<arg>-AlegacyConfigRoot=true</arg>
</compilerArgs>
</configuration>
</plugin>
</plugins>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,13 @@

import java.util.Map;

import org.eclipse.microprofile.config.ConfigProvider;
import jakarta.inject.Inject;

import org.testcontainers.couchbase.CouchbaseContainer;
import org.testcontainers.couchbase.CouchbaseService;

import com.couchbase.quarkus.extension.runtime.CouchbaseConfig;

import io.quarkus.deployment.IsNormal;
import io.quarkus.deployment.annotations.BuildStep;
import io.quarkus.deployment.annotations.BuildSteps;
Expand All @@ -37,6 +40,9 @@ public class CouchbaseDevService {

static volatile RunningDevService devService;

@Inject
CouchbaseConfig config;

@BuildStep
DevServicesResultBuildItem startCouchBase(
CuratedApplicationShutdownBuildItem closeBuildItem) {
Expand All @@ -52,12 +58,8 @@ DevServicesResultBuildItem startCouchBase(
}

private QuarkusCouchbaseContainer startContainer() {
String userName = ConfigProvider.getConfig()
.getOptionalValue("quarkus.couchbase.username", String.class).orElse("Administrator");
String password = ConfigProvider.getConfig()
.getOptionalValue("quarkus.couchbase.password", String.class).orElse("password");
String version = ConfigProvider.getConfig().getValue("quarkus.couchbase.version", String.class);
QuarkusCouchbaseContainer couchbase = new QuarkusCouchbaseContainer(version, userName, password);
QuarkusCouchbaseContainer couchbase = new QuarkusCouchbaseContainer(config.version(), config.username(),
config.password());
couchbase.start();
return couchbase;
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
/*
* Copyright (c) 2025 Couchbase, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.couchbase.quarkus.extension.deployment;

import com.couchbase.client.core.util.ConnectionString;
import com.couchbase.quarkus.extension.runtime.CouchbaseConfig;

import io.quarkus.deployment.IsDevelopment;
import io.quarkus.deployment.annotations.BuildStep;
import io.quarkus.devui.spi.page.CardPageBuildItem;
import io.quarkus.devui.spi.page.Page;

public class CouchbaseDevUiProcessor {
@BuildStep(onlyIf = IsDevelopment.class)
public CardPageBuildItem pages(CouchbaseConfig config) {
CardPageBuildItem cardPageBuildItem = new CardPageBuildItem();

var hostname = extractHostnameFromConnectionString(config.connectionString());
var clusterUiUrl = "http://" + hostname + ":8091/ui/index.html";

String JAVA_SDK_DOCS = "https://docs.couchbase.com/java-sdk/current/hello-world/overview.html";

cardPageBuildItem.addPage(Page.externalPageBuilder("Cluster Dashboard")
.url(clusterUiUrl, clusterUiUrl)
.doNotEmbed()
.icon("font-awesome-solid:database"));

cardPageBuildItem.addPage(Page.externalPageBuilder("Java SDK Docs")
.url(JAVA_SDK_DOCS, JAVA_SDK_DOCS)
.doNotEmbed()
.icon("font-awesome-solid:couch"));

cardPageBuildItem.addPage(Page.externalPageBuilder("Extension Guide")
.url("https://docs.quarkiverse.io/quarkus-couchbase/dev/index.html")
.isHtmlContent()
.icon("font-awesome-solid:book"));

return cardPageBuildItem;
}

/**
* Extracts the first hostname from the connection string to redirect to the Cluster UI Dashboard.
*
* @param connectionString The connection string specified in application.properties.
* @return The first hostname.
*/
private String extractHostnameFromConnectionString(String connectionString) {
ConnectionString connStr = ConnectionString.create(connectionString);
return connStr.hosts().get(0).host();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@
*/
package com.couchbase.quarkus.extension.deployment;

import java.util.Optional;

import jakarta.enterprise.context.ApplicationScoped;

import com.couchbase.client.core.api.kv.CoreKvBinaryOps;
import com.couchbase.client.core.api.kv.CoreKvOps;
import com.couchbase.client.java.Cluster;
import com.couchbase.quarkus.extension.runtime.CouchbaseConfig;
import com.couchbase.quarkus.extension.runtime.CouchbaseRecorder;
Expand All @@ -26,21 +30,120 @@
import io.quarkus.deployment.annotations.BuildStep;
import io.quarkus.deployment.annotations.ExecutionTime;
import io.quarkus.deployment.annotations.Record;
import io.quarkus.deployment.builditem.nativeimage.NativeImageConfigBuildItem;
import io.quarkus.deployment.builditem.nativeimage.NativeImageProxyDefinitionBuildItem;
import io.quarkus.deployment.builditem.nativeimage.ReflectiveClassBuildItem;
import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
import io.quarkus.runtime.metrics.MetricsFactory;
import io.quarkus.smallrye.health.deployment.spi.HealthBuildItem;

public class CouchbaseProcessor {

@BuildStep
@Record(ExecutionTime.RUNTIME_INIT)
public void produceCouchbaseClient(CouchbaseRecorder recorder,
CouchbaseConfig config,
Optional<MetricsCapabilityBuildItem> metricsCapability,
BuildProducer<SyntheticBeanBuildItem> syntheticBeans) {

var metricsEnabled = config.metricsEnabled()
&& metricsCapability.isPresent()
&& metricsCapability.get().metricsSupported(MetricsFactory.MICROMETER);

syntheticBeans.produce(SyntheticBeanBuildItem
.configure(Cluster.class)
.scope(ApplicationScoped.class)
.unremovable()
.supplier(recorder.getCluster(config))
.supplier(recorder.getCluster(config, metricsEnabled))
.setRuntimeInit()
.done());
}

@BuildStep
HealthBuildItem addHealthCheck(CouchbaseConfig couchbaseConfig) {
return new HealthBuildItem("com.couchbase.quarkus.extension.runtime.health.CouchbaseReadyCheck",
couchbaseConfig.healthEnabled());
}

@BuildStep
NativeImageProxyDefinitionBuildItem kvProxies() {
return new NativeImageProxyDefinitionBuildItem(
CoreKvOps.class.getName(),
CoreKvBinaryOps.class.getName());
}

@BuildStep
NativeImageConfigBuildItem nativeConfiguration() {
NativeImageConfigBuildItem.Builder builder = NativeImageConfigBuildItem.builder();
// NettyProcessor configures UnpooledByteBufAllocator and other related Netty classes for runtime initialization,
// but an instance of it is making its way into the native heap through CoreHttpRequest$Builder.EMPTY_BYTE_BUFF.
// Therefore, we defer its initialization until runtime.
builder.addRuntimeInitializedClass("com.couchbase.client.core.endpoint.http.CoreHttpRequest$Builder");
return builder.build();
}

@BuildStep
ReflectiveClassBuildItem reflection() {
return ReflectiveClassBuildItem.builder(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised/confused at many of these things needing reflective access. Like CoreConflictResolutionType - it doesn't seem to be used in any reflective way, anywhere in the code base.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're not used in the extension if that's what you mean, but these are all used in a normal functioning app with the Java SDK.
Quarkus apps can add annotations i.e. @RegisterForReflection to their own code, but not to third-party libraries like ours so we need to provide it ourselves.

If we don't register them for reflection here, an app built with the extension will fail to serialize/deserialize these classes as it didn't store the reflection info on their fields or methods.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please forgive my ignorance. Why does a class like CoreConflictResolutionType need to be serialized/deserialized?

Copy link
Contributor Author

@emilienbev emilienbev Jan 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this one it's a field in CoreBucketSettingsJson, which is created in ClassicCoreBucketManager.getAllBuckets()for example. That gives the bucket from the http response as a JsonNode to the ObjectMapper: Mapper.convertValue(node, CoreBucketSettingsJson.class);.

In the native image, (in my understanding) the information about the class and its fields/methods which would usually be retrieved via reflection isn't saved (or maybe stripped out), so the ObjectMapper would not be able to match the classes' field names or types to construct the object.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we'll need to update this list every time we do Jackson data binding with a new class? :-/

How does this work with Jackson in general? Like, is there a Quarkus extension for Jackson that knows to preserve reflection info for any type used in an objectMapper.convertValue(Object, Class) statement? Maybe we could use a similar technique to mark Mapper.convertValue?

In the shorter term, would it be simple to preserve reflection info for all classes in a package? We'd still need to maintain the list of packages, but perhaps updates would be less frequent.

(I'm not suggesting this issue should block the merge; just trying to identify areas that could be improved in the future.)

Copy link
Contributor Author

@emilienbev emilienbev Jan 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we'll need to update this list every time we do Jackson data binding with a new class?

Yes indeed. This list was built by running FIT, and seeing when an reflection error would happen and which operation it affected.

Like, is there a Quarkus extension for Jackson that knows to preserve reflection info for any type used in an objectMapper.convertValue(Object, Class) statement?

For Jackson data-binding this ReflectiveClassBuildItem() from Quarkus generates a reflection-config.json that is passed to Graal, which allows us to target classes in third party dependencies. If those classes were in a Quarkus app or directly in this project, we could use annotations such as @RegisterForReflection on the class to do so.
Some other reflection APIs are automatically analyzed by Graal and do not need manual configuration (link to docs).

There is a Jackson extension for Quarkus that does extra processing and produces an ObjectMapper synthetic bean like for our Cluster, but we would have to explicitly use it in the SDK and configure it too. I haven't gone down that path as it seemed it wasn't needed for us.

would it be simple to preserve reflection info for all classes in a package?

That's interesting, I guess so? It would increase the size of the native-image, I'm not sure by how much.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation @emilienbev . Tbh I continue to be rather horrified by the potential maintenance burden of this extension (I know you've put a lot of work into that already), and the optimise-at-any-maintenance-cost philosophy taken by Graal and its closed-system assumptions.

Can we schedule some time to look at solutions, particularly that suggestion of just including all reflection info? I would really like to get this extension safe and easily maintainable first (plus have it not be such a time sink for you), and look at what-I-suspect-will-be-micro optimisations later, if there is uptake that warrants it.

Happy to help if needed, though I think I will just get in your way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep would love to chat with you on this. I could use your input on both maintaining this extension, and maintaining the FIT performer which does not tolerate change well.

new String[] {
//Transactions
"com.couchbase.client.core.util.ReactorOps",
//Error
"com.couchbase.client.core.transaction.components.ActiveTransactionRecordEntry",
"com.couchbase.client.core.transaction.components.DocRecord",
//Bucket Manager
"com.couchbase.client.core.manager.bucket.CoreCompressionMode",
"com.couchbase.client.core.manager.bucket.CoreEvictionPolicyType",
"com.couchbase.client.core.manager.bucket.CoreStorageBackend",
"com.couchbase.client.core.manager.bucket.CoreConflictResolutionType",
"com.couchbase.client.core.manager.bucket.BucketSettings",
"com.couchbase.client.java.manager.bucket.BucketSettings",
"com.couchbase.client.core.manager.bucket.CoreBucketSettings",
"com.couchbase.client.core.classic.manager.CoreBucketSettingsJson",
"com.couchbase.client.core.config.BucketType",
"com.couchbase.client.core.msg.kv.DurabilityLevel",

"com.couchbase.client.core.config.CollectionsManifestCollection",
"com.couchbase.client.core.config.CollectionsManifestScope",
"com.couchbase.client.core.config.CollectionsManifest",
"com.couchbase.client.core.error.ErrorCodeAndMessage",
"com.couchbase.client.core.msg.BaseResponse",
"com.couchbase.client.core.endpoint.http.CoreHttpResponse",
"com.couchbase.client.core.endpoint.http.CoreCommonOptions",
"com.couchbase.client.core.logging.RedactableArgument",
"com.couchbase.client.core.msg.CancellationReason",
"com.couchbase.client.core.api.manager.search.CoreSearchIndex",
"com.couchbase.client.java.manager.search.SearchIndex",
//Search
//Result
"com.couchbase.client.core.api.search.result.CoreAbstractSearchFacetResult",
"com.couchbase.client.core.api.search.result.CoreDateRangeSearchFacetResult",
"com.couchbase.client.core.api.search.result.CoreNumericRangeSearchFacetResult",
"com.couchbase.client.core.api.search.result.CoreReactiveSearchResult",
"com.couchbase.client.core.api.search.result.CoreSearchDateRange",
"com.couchbase.client.core.api.search.result.CoreSearchFacetResult",
"com.couchbase.client.core.api.search.result.CoreSearchMetrics",
"com.couchbase.client.core.api.search.result.CoreSearchNumericRange",
"com.couchbase.client.core.api.search.result.CoreSearchResult",
"com.couchbase.client.core.api.search.result.CoreSearchRow",
"com.couchbase.client.core.api.search.result.CoreSearchRowLocation",
"com.couchbase.client.core.api.search.result.CoreSearchRowLocations",
"com.couchbase.client.core.api.search.result.CoreSearchStatus",
"com.couchbase.client.core.api.search.result.CoreSearchTermRange",
"com.couchbase.client.core.api.search.result.CoreTermSearchFacetResult",
//Facet
"com.couchbase.client.core.api.search.facet.CoreDateRange",
"com.couchbase.client.core.api.search.facet.CoreDateRangeFacet",
"com.couchbase.client.core.api.search.facet.CoreNumericRange",
"com.couchbase.client.core.api.search.facet.CoreNumericRangeFacet",
"com.couchbase.client.core.api.search.facet.CoreSearchFacet",
"com.couchbase.client.core.api.search.facet.CoreTermFacet",
//Vector
"com.couchbase.client.core.api.search.vector.CoreVector",
"com.couchbase.client.core.api.search.vector.CoreVectorQuery",
"com.couchbase.client.core.api.search.vector.CoreVectorQueryCombination",
"com.couchbase.client.core.api.search.vector.CoreVectorSearch",
"com.couchbase.client.core.api.search.vector.CoreVectorSearchOptions"
}).fields().methods().build();
}
}
Loading
Loading