Spring Boot 3.0 (RC2) Observability Questions

[goafabric]

Intro
I've been working with spring sleuth Zipkin in Spring Boot 2.x in Combination with Jaeger
This helped me to get connected Spans between two Spring Boot Applications, via RestTemplate
Now wit Boot 3.0 i have some puzzeling questions i cannot figure out.

I already read these two articles/documentation:

• https://docs.spring.io/spring-boot/docs/3.0.0-SNAPSHOT/reference/html/actuator.html#actuator.micrometer-tracing
  -https://spring.io/blog/2022/10/12/observability-with-spring-boot-3

As well as reading the code
But as I am now stuck for 2 weeks, it would be nice if someone can help me out here.

As dependencies I am using the followling, but also tried the "otel" variants

implementation 'io.micrometer:micrometer-tracing-bridge-brave'
implementation 'io.zipkin.reporter2:zipkin-reporter-brave'

Question 1
With Spring Security enabled (and Cicrcuitbreaker) a lot of additional Info concerning the Filterhchains is added. (Image attached)
Is there a way to disable this ?

Question2
I "guess" in AOT mode observability has become a compile time property. (got this from the latest Devoxx Video)
In the veins of it's either enabled or disabled during compile time and not modifiable during runtime ?

Question 3
While with Sleuth, Boot 2.x + Resttemplate i get interconnected spans between my two applications.
This is not working with Boot 3.x ... I just get 2 seperated spans for each of my Applications.
Following the documentation above it is a little unclear for me if resttemplate will automatically get instrumented
and create the spans .. like it is was done in the past .. or not.

Thank you

[snicoll]

@goafabric as mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements. Thanks for trying the RC but asking questions here isn't the right thing to do. Please chat with the community on Gitter or ask your questions on StackOverflow.

[mhalbritter]

While with Sleuth, Boot 2.x + Resttemplate i get interconnected spans between my two applications. This is not working with Boot 3.x ... I just get 2 seperated spans for each of my Applications.

You mean with Sleuth, you got a trace spanning both applications and with SB3 this doesn't work? If so, this sounds like a bug.

[wilkinsona]

While questions don't belong in the issue tracker, I am going to re-open this as it's not clear to me if there's a bug here of it the documentation's lacking.

Following the documentation above it is a little unclear for me if resttemplate will automatically get instrumented
and create the spans .. like it is was done in the past .. or not.

The documentation describes the automatic instrumentation of RestTemplate and Marcin's blog post shows this in action. If it isn't working for you and you would like us to spend some time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

[goafabric]

@wilkinsona
thx i will take a 2nd look and report back with more details

[goafabric]

Description
Ok with the help of @wilkinsona i was able to identify the problem in 5 minutes, after searching for weeks.
One would say now ... hey that was obvious from the beginning .. well for me a long term spring boot user it was not.

Let me explain:
First and foremost i now know that the documentation can be found here
https://docs.spring.io/spring-boot/docs/3.0.0-RC2/reference/htmlsingle/#actuator.metrics.supported.http-clients
ant not here
https://docs.spring.io/spring-boot/docs/3.0.0-SNAPSHOT/reference/html/actuator.html#actuator.micrometer-tracing

Two different versions and two different sections ... but it think that is partly the price of using a non GA version.
Then while the documentation states what has to be done
"For that, you have to inject the auto-configured builder and use it to create instances:"

That is stated very "weak" .. at least if you are used to Spring Boot 2.x
Marcin's Post however addresses the issue more strongly also with an example .. which helps.

So what did I do up until now
@Bean public RestTemplate restTemplate() { return new RestTemplateBuilder().build(); }

What actually has to be done
@Bean RestTemplate restTemplate(RestTemplateBuilder builder) { return builder.build(); }

Yes stated .. and yes .. now it all makes sense .. because creating a "new RestTemplateBuilder"
can presumeably never get instrumented.

But Option number 1 worked perfectly finde in Boot 2.7.5 + sleuth ...

Proposal
I am not here to debate .. and I am very thankful to Andy for pointing out the solution.
What I would recommend that the final documentation has a small code example as above.
And also states eplicitely that the old way does not work any more ..

[goafabric]

So I think the biggest point is solved now for me.
But there is still Question 1+2 .. which is just that a question...
Is there also documentation somwhere that describes the possible configuration
of SecurityFitlerChain beeing part of the trace or not ?

[wilkinsona]

That is stated very "weak" .. at least if you are used to Spring Boot 2.x

As documented here, the same was true in Boot 2.x if you wanted metrics from RestTemplate. Documenting differences in Sleuth's behaviour is out of scope for Boot's reference documentation. I think it would be better covered in the Sleuth Migration Guide that @marcingrzejszczak is working on.

Is there also documentation somwhere that describes the possible configuration of SecurityFitlerChain beeing part of the trace or not?

Spring Security's documentation describes how to configure Spring Security's observability, including a section on disabling it.

[goafabric]

ok thank you

[bclozel]

Spring Security's documentation describes how to configure Spring Security's observability, including a section on disabling it.

In the context of a Spring Boot application, I think contributing an ObservationPredicate directly would work as well. Also, I think that all predicates must match for an observation to be enabled. So it looks like the code snippet in the Spring Security reference documentation does not disable it (is it missing a ! in the predicate?).

I've added a new section on that in our migration docs.

[marcingrzejszczak]

As documented here, the same was true in Boot 2.x if you wanted metrics from RestTemplate. Documenting differences in Sleuth's behaviour is out of scope for Boot's reference documentation. I think it would be better covered in the Sleuth Migration Guide that @marcingrzejszczak is working on.

In the wiki you can read this

The default Tracer now is OpenTelemetry tracer, the default Trace Id format is 128 bit and default Context Propagation format is W3C. By default we don't support joined spans.

joined spans are exactly the thing that you have the same span on the receiver and sender side. I will put a better explanation there in the wiki

[goafabric]

Disabling works in the case of
name.startsWith("http.server.")
but not in case of

name.startsWith("spring.security")

I guess the expected return type of spring security is not Compatible with the Noop.

java.lang.ClassCastException: class io.micrometer.observation.Observation$Context cannot be cast to class org.springframework.security.web.ObservationFilterChainDecorator$FilterChainObservationContext (io.micrometer.observation.Observation$Context and org.springframework.security.web.ObservationFilterChainDecorator$FilterChainObservationContext are in unnamed module of loader 'app') at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:179) ~[spring-security-web-6.0.0-RC2.jar:6.0.0-RC2]

[bclozel]

@goafabric Could you raise an issue on Spring Security for that please?

[goafabric]

@bclozel
sure ...
spring-projects/spring-security#12268

[goafabric]

one thing concerning the migration guide,
if not covered yet, please mention the CHANGE from B3 -> W3C

Because when combining boot 2.x with sleuth + 3.x with micrometer
spans will only connect when setting:

management.tracing.propagation.type: "B3"

I think the same thing will be true when using istio

[bclozel]

@marcingrzejszczak What do you think about the comment above?

[marcingrzejszczak]

It's written there in the same sentence that we have migrated from B3 to W3C

The default Tracer now is OpenTelemetry tracer, the default Trace Id format is 128 bit and default Context Propagation format is W3C. By default we don't support joined spans (this means that when you have e.g. an HTTP span, you will no longer see the same span being there on the client and sender side, you will see two separate spans now).

I will however write how to do the migration. You should set in Boot 2.x app spring.sleuth.propagation.type=w3c,b3 . That way you will propagate both w3c and b3 and then in Boot 3.x you don't need to change anything. I will update the migration guide.