Quarkus 3 OpenTelemetry hardening

[michalvavrik]

Summary

• OpenTelemetry JDBC instrumentation added to Narayana Transactions module as we already test there REST traces and start Jaeger:
  • tests Use a property to enable OpenTelemetry for JDBC and use the OpenTelemetry datasource instead of the OpenTelemetry driver quarkusio/quarkus#31567, covers scenarios missing in upstream test coverage:
    • works in native (also) for MySQL and MSSQL datasource, important for we need to verify their drivers in native
    • use actual Jaeger container, upstream uses InMemorySpanExporter or mocks everywhere
    • test instrumentation in OpenShift environment with Red Hat registry images
  • tests Fix vertx otel sdk reload in devmode quarkusio/quarkus#28792 as upstream doesn't test this at all
    • in order to verify this issue, module sql-db/narayana-transactions is now using RESTEasy Reactive, which is preferred REST stack anyway and not relevant for Narayana tests
• Smallrye Context Propagation with OpenTelemetry is also placed in Narayana Transactions module as there we already have (transitive) Smallrye Propagation Context dependency and running tests so we don't have to start extra application for that (this way, impact on resources is minimal)
• Quarkus Scheduler tracing because upstream only test this with unit test and doesn't even check traces go to exporter
• OTEL beans available according to the MicroProfile Telemetry Tracing specification, it's important as upstream:
  • only use @QuarkusTest so we want to check normal lifecycle and native mode
  • only tests static OTEL methods returns same stuff as injected (so basically tests used class loader is correct), but we are actually going to use them little bit and see that traces are propagated to Jaeger
  • I absolutely failed to find reasonable usage of injected Baggage. It is immutable and I can't see to be able to propagate anything in.I can see that OTEL java instrumentation git project basically use it as carrier within span context, but then you can't get it into injected immutable baggage anyway. Upstream docs doesn't mention baggage anyway and doesn't support it for MDC logging. Perhaps I needed to use Jaeger OTEL propagator...
• RESTEasy Classic and Reactive now have filters that adds 2 of recommended code attributes
  • it's probably not super important feature, but I thought it would be nice to have at least smoke test for there filters (in native) and verification was added to other test so it won't cost us more than couple of seconds
• OpenTelemetry SDK autoconfiguration has now limited support for those OTEL configuration properties we already support
  • upstream PR was massive, however it's importance is mostly for future changes, right now from user point of view, it is basically cutting down quarkus.otel.some-property-name to otel.some-property-name https://github.com/quarkusio/quarkus/blob/43a53588daa126477e24e81c75abfbf017547df5/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/OpenTelemetryProducer.java#L143 and passing them to SDK builder, so what we want is to test that selected properties are passed to OTEL (otherwise we would be expensively testing whether someone didn't do typo)
    • we test that quarkus.otel.service.name has preference over quarkus.application.name and it doesn't OpenTelemetry SDK autoconfiguration ignores OTEL service name in favor of Quarkus app name quarkusio/quarkus#33317 (so I opened issue)
    • we test that quarkus.otel.attribute.value.length.limit actually limits attribute length. that is OTEL business, but we want to ensure Quarkus OpenTelemetry extensions passed the property
  • important change is that quarkus.opentelemetry has changed to quarkus.otel, I didn't migrate properties as @linartova is working on this and backwards compatibility should work for the moment being (and changes are not one to one, e.g. quarkus.opentelemetry.tracer.exporter.otlp.endpoint has changed to quarkus.otel.exporter.otlp.traces.endpoint as documented in migration guide)

Please select the relevant options.

• Bug fix (non-breaking change which fixes an issue)
• Dependency update
• Refactoring
• Backport
• New scenario (non-breaking change which adds functionality)
• This change requires a documentation update
• This change requires execution against OCP (use run tests phrase in comment)

Checklist:

• Methods and classes used in PR scenarios are meaningful
• Commits are well encapsulated and follow the best practices

[michalvavrik]

run tests

[rsvoboda]

One question, otherwise this looks really nice!