Initial Observability extension #36470

alesj · 2023-10-13T12:15:35Z

This adds dynamic lookup of possible observability dev resources / services.
It also adds REST clients for PromQL (Prometheus query language) and VictoriaMetrics REST api support; push, etc

The dev resources / services can be run in 3 different ways

re-usable dev services (default)
simple start/stop dev services
single / plain per QuarkusTestResource dev resource

(1) can be configured
(2) depends what's available on the classpath
(3) needs to be explicitly used in QuarkusTestResource

Currently supported dev services

Grafana
Jaeger
OTel
VictoriaMetrics DB
VictoriaMetrics Agent / scraper

It should be easy to add new / other observability dev services in the future,
e.g. currently missing Prometheus, etc
by adding a new DevResourceLifecycleManager implementation.

Current TODO list

test with 2 app running at the same time, propagating tracing to same dev service(s)
Grafana-all-in-one Docker image support
Prometheus dev service
reload and other (combo) tests

brunobat · 2023-10-17T15:53:06Z

...ility/common/src/main/java/io/quarkus/observability/common/config/VictoriaMetricsConfig.java

+
+@ConfigGroup
+@DevTarget("io.quarkus.observability.devresource.victoriametrics.VictoriaMetricsResource")
+public interface VictoriaMetricsConfig extends ContainerConfig {


I know that the project started with VictoriaMetrics, but I would remove this for a later PR. It adds to much complexity now.
We wouldn't really need PromQL and all it's needed to support it.

Afais, Prometheus is up high there in the wish list, and VM == Prometheus.
I'm planning on adding real Prometheus asap.

And as such, I would leave PromQL -- it might come useful to some users.
Might probably need some more tests, but REST API was "copy/pasted" (and implemented) from the docs.

I'm not contesting being here at some point, I'm just concerned about the size of the PR. We need to break this up because the important parts are totally obfuscated.
I strongly believe we should have only Jaeger-all and Grafana-all on this first PR.
Also believe that expanding the number of destinations is secondary and we should focus on making sure the user experience is spot on.

brunobat · 2023-10-17T15:54:46Z

extensions/observability/testlibs/devresource-grafana/pom.xml

+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <artifactId>quarkus-observability-testlibs</artifactId>


Rename from testlibs to observability-devresource?

There might be other test libs (artifacts) in the future, not just devresources ...

It's just because the name is very misleading... It's not just for tests, I imagine.

Yeah. OK, will try to think of a better name.

brunobat · 2023-10-17T15:58:36Z

integration-tests/observability-multiapp/src/main/resources/application.properties

+quarkus.micrometer.export.otlp.enabled=true
+quarkus.micrometer.export.otlp.publish=true
+quarkus.micrometer.export.otlp.default-registry=true
+quarkus.micrometer.export.otlp.url=http://${quarkus.otel-collector.http}/v1/metrics


Should it probably be the other way around? Instead of quarkus.otel-collector.http, should we pick up the value from quarkus.micrometer.export.otlp.url

Hmmm, afais, no -- you need the location / root url of OTel Collector.

brunobat · 2023-10-17T16:02:21Z

integration-tests/observability/pom.xml

+        <dependency>
+            <groupId>io.quarkus</groupId>
+            <artifactId>quarkus-observability-victoriametrics</artifactId>
+        </dependency>
+        <!-- Add these deps as non-test / plain dependency, so we can run dev mode to check -->
+        <dependency>
+            <groupId>io.quarkus</groupId>
+            <artifactId>quarkus-observability-devresource-victoriametrics</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.quarkus</groupId>
+            <artifactId>quarkus-observability-devresource-grafana</artifactId>
+        </dependency>


I think we should simplify the setup for the users. Having just quarkus-observability already including a default devservice would be optimal.
We could create a different artifact for other options... But just a single dependency.

Yeah, it's on my todo list - as we discussed - I'll add an "uber" pom, with all "normal" dependecies.
e.g. all probably don't make sense, as some exclude each other

brunobat · 2023-10-20T08:55:33Z

I'd suggest to reduce the PR size so we can better iron out how users are going to use this.

To maintain the user choice of runtime extensions, we could rename quarkus-observability to quarkus-observability-devservices leaving the quarkus-observability to be used in the future to aggregate runtime metrics, traces, and logs.
we should have implementations per output. Ex: quarkus-observability-jaeger, quarkus-observability-grafana, quarkus-observability-victoriametrics-otlp (service->otel collector->victoria), quarkus-observability-victoriametrics-agent and quarkus-observability-victoriametrics-service.
It theory we can do this in 3 PRs+.
For victoria metrics we can have setup variations.

quarkus-bot · 2024-01-10T10:41:03Z

Failing Jobs - Building `0f5c776`

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	JVM Tests - JDK 17	`Build`	Failures	Logs	Raw logs	🔍
✖	JVM Tests - JDK 21	`Build`	Failures	Logs	Raw logs	🔍

Full information is available in the Build summary check run.
You can consult the Develocity build scans.

Failures

⚙️ JVM Tests - JDK 17 #

- Failing: integration-tests/observability-multiapp

📦 integration-tests/observability-multiapp

✖ io.quarkus.observability.test.SharedTracingTest.testTracing line 14 - More details - Source on GitHub

java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)

⚙️ JVM Tests - JDK 21 #

- Failing: integration-tests/observability-multiapp

📦 integration-tests/observability-multiapp

✖ io.quarkus.observability.test.SharedTracingTest.testTracing line 14 - More details - Source on GitHub

java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:589)
	at java.base/sun.nio.ch.Net.connect(Net.java:578)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:751)
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)

alesj · 2024-01-22T10:29:16Z

Closing in favor of a new, similar PR, but (hopefully) lighter version.

quarkus-bot bot added area/core area/dependencies Pull requests that update a dependency file area/devtools Issues/PR related to maven, gradle, platform and cli tooling/plugins area/documentation labels Oct 13, 2023

alesj force-pushed the obs1 branch from 7c0fea5 to 93a6d8e Compare October 13, 2023 19:05