Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry instrumentation stops working for resteasy-reactive resources and JDBC after dev mode reload #29645

Closed
famod opened this issue Dec 2, 2022 · 7 comments · Fixed by #28792

Comments

@famod
Copy link
Member

famod commented Dec 2, 2022

Describe the bug

OTEL instrumentation seems to stop working for resteasy-reactive resources and JDBC calls when dev mode is reloaded (manually or due to a file change).
The reproducer further down demonstrates that via the logged MDC values, but it is not a mere MDC issue.
I first found this issue on my local Jaeger instance. I was missing some traces and spans and instead I got some unexpected traces that are supposed to be sub-spans of traces.
I also didn't see any JDBC sub-spans in my @Scheduled traces anymore (see also #28552 (comment)).

Expected behavior

Traces are generated just like before the dev mode reload

Actual behavior

No proper traces are generated anymore after dev mode reload

How to Reproduce?

  1. start dev mode in the OTEL-quickstart via:
    mvn quarkus:dev -Dquarkus.log.console.format='%d{HH:mm:ss} %-5p (%X{traceId},%X{spanId}) [%c{2.}] (%t) %s%e%n'
    
  2. call http://localhost:8080/hello
  3. verify that trace and span ids are logged, e.g. (first bracket):
    17:25:09 INFO  (f67e99e80328a210c23f58665ade8ffd,e561a98c8dcc9b37) [or.ac.op.TracedResource] (executor-thread-1) hello
    
  4. hit s in dev mode (or change a file and call /hello)
  5. call http://localhost:8080/hello
  6. from this point on only zeros are logged for both ids:
    17:25:47 INFO  (00000000000000000000000000000000,0000000000000000) [or.ac.op.TracedResource] (executor-thread-0) hello
    

Note: This quickstart is a bit sparse ATM, see also: quarkusio/quarkus-quickstarts#1120

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.14.2

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

Another user had the same issue a few months back: #26804 (comment)

It's worth stressing that bean methods annotated with @WithSpan are still working after reload, but obviously, if those calls are sub-spans of REST calls they are shown as separate traces in Jaeger as there is no parent trace anymore.

@famod famod added the kind/bug Something isn't working label Dec 2, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Dec 2, 2022

@geoand
Copy link
Contributor

geoand commented Dec 5, 2022

@brunobat weren't you working on something related?

@radcortez
Copy link
Member

There is a PR for it: #28792

@famod
Copy link
Member Author

famod commented Dec 5, 2022

Nice, thanks!

@gsmet
Copy link
Member

gsmet commented Dec 5, 2022

@famod is it a regression in 2.14? I'm wondering if I should backport it for the next 2.14 (planned for tomorrow).

@radcortez
Copy link
Member

No. It never worked :)

@famod
Copy link
Member Author

famod commented Dec 5, 2022

It's a rather confusing issue. First thing you probably do when you're new to OTEL is to play around with it in dev mode. So after a reload you are faced with partially broken tracing and as a newbie you start looking for your own mistakes (but there are none here).
Long story short, I do see value in backporting this to more releases than "just" 2.15.
For me personally it doesn't make much of a difference since I'm probably migrating to 2.15 sooner than later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants