ClickHouse HTTP client instrumentation does not work when no database selected

[devurandom]

Describe the bug

OpenTelemetry JavaAgent 2.6.0 should add instrumentation to the ClickHouse JavaAgent. I tried it with our application (Java 11, Clojure 1.11.3, ClickHouse HTTP Client 0.6.2, using the built-in Java HTTP Client). As with previous JavaAgent versions, I only see the spans created by io.opentelemetry.java-http-client.

Steps to reproduce

1. Start an OpenTelemetry stack on http://localhost:4317.
2. Follow the instructions in https://github.com/devurandom/opentelemetry-java-instrumentation-clickhouse-clojure-mwe to run the sample application.
3. Observe a HTTP client span with name POST, but no ClickHouse client span.

I looked at the traces using OpenTelemetry Collector ingesting into Grafana Tempo, and using https://github.com/dotnet/aspire/tree/main/src/Aspire.Dashboard locally, but both only show the io.opentelemetry.java-http-client spans.

Expected behavior

I expected to see two nested spans, one created by the ClickHouse client instrumentation, the other created by the HTTP client instrumentation.

Actual behavior

I only see a HTTP client span with name POST, but no ClickHouse client span.

Javaagent or library instrumentation version

v2.6.0

Environment

JDK:

❯ java --version
openjdk 21.0.3 2024-04-16
OpenJDK Runtime Environment (Red_Hat-21.0.3.0.9-1) (build 21.0.3+9)
OpenJDK 64-Bit Server VM (Red_Hat-21.0.3.0.9-1) (build 21.0.3+9, mixed mode, sharing)

OS:

❯ grep PRETTY /etc/os-release
PRETTY_NAME="Fedora Linux 40 (KDE Plasma)"

Additional context

This continues #11660 (comment).

[jaydeluca]

I believe the issue here is the instrumentation is not properly handling the case where a database name is not passed in via the connection string.

I was able to reproduce the issue by running:

./bin/start http://localhost:8123

I then identified the exception stack trace:

[otel.javaagent 2024-07-18 07:07:16:854 -0400] [main] DEBUG io.opentelemetry.javaagent.bootstrap.ExceptionLogger - Failed to handle exception in instrumentation for com.clickhouse.client.ClickHouseClient
java.util.NoSuchElementException: No value present
        at java.base/java.util.Optional.get(Optional.java:143)
        at com.clickhouse.client.ClickHouseClient.executeAndWait(ClickHouseClient.java:875)
        at otel_ch_clj_mwe$execute_BANG_.invokeStatic(otel_ch_clj_mwe.clj:36)
        at otel_ch_clj_mwe$execute_BANG_.invoke(otel_ch_clj_mwe.clj:33)
        at otel_ch_clj_mwe$_main.invokeStatic(otel_ch_clj_mwe.clj:42)
        at otel_ch_clj_mwe$_main.doInvoke(otel_ch_clj_mwe.clj:38)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.main$main_opt.invokeStatic(main.clj:514)
        at clojure.main$main_opt.invoke(main.clj:510)
        at clojure.main$main.invokeStatic(main.clj:664)
ure.main$main.doInvoke(main.clj:616)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.main.main(main.java:40)

And by adding debug logs into the instrumentation identified that it is the getDatabase().get() method throwing the NoSuchElementException here

I then tried again with the database included in the connection string:

./bin/start http://localhost:8123/default

And everything worked as expected:

I will work on a fix for this

[devurandom]

Fix confirmed: Instrumentation of ClickHouse Java HTTP Client 0.6.2 with OpenTelemetry Java Agent 2.7.0 gives me insight into ClickHouse operations above the HTTP level and creates spans with db.system="clickhouse", db.statement and other db.* attributes for ClickHouse requests.
Thanks!