Diagnostics is not setting the Activity ParentId for OpenTelemetry tracing

[rineshpk]

Library name and version

Azure.Messaging.ServiceBus 7.9.0

Query/Question

When receiving a message through the Azure Service Bus, the parent id for the current activity is null and I could see the id received in the metadata in the 'Diagnostic-Id' field is set as a Link for the current activity. When I export the traces, it is not producing the correct trace chain (exported as two separate traces for send and receive).

If we create one activity manually and set the 'Diagnostic-Id' to the parent id on the receiver side, exported traces chain is displayed in the correct hierarchy(send and the custom activity created at the receiver side in a single trace).

Why diagnostics extension is not setting the parent id and what is the use of Link if it's not useful in chaining the trace?

Environment

No response

[jsquire]

@lmolkova : You may have the best perspective on this. Would you be so kind as to share your thoughts?

[JoshLove-msft]

@lmolkova, is this dependent on the App Insights fix here - microsoft/ApplicationInsights-dotnet#2593?
How would customers consume the fix?

[rineshpk]

@JoshLove-msft I don't think it depends on App Insight fix as I'm not using App Insight here. I've integrated OpenTelemetry/ActivitySource (as in described here ) and exported the traces to Jaeger.

[lmolkova]

@rineshpk

Links is OpenTelemetry concept that allows connecting spans without introducing parent-child relationship between them. This is necessary in batching scenarios - they're recommended by OTel messaging spec.

Jaeger supports links to some extent - you should be able to see linked traces when looking into consumer (processing) spans:

[rineshpk]

@lmolkova Thank you! Yes, I’m able see the reference link in Jaeger. But, my use case is something different and I would like to see the message send/receive in a single trace (as in the case of setting the Diagnostic-Id as parentId).

And, I could see the MessagingDiagnosticExtension class in the deprecated Microsoft.Azure.ServiceBus package was setting the Diagnostic-Id to the parentId.

Is there a way to accomplish this parent-child relationship with Azure.Messaging.ServiceBus instrumentation or is this something not supported anymore?

[JoshLove-msft]

The new library uses links to connect the individual message activities with the parent send activity, as per the OTel messaging spec linked above. This was a conscious decision and a departure from the previous library that set diagnostic-Id to the parent activity. As far as I can tell, all of the relationships are there, but it may be that Jaeger does not support visualizing the relationships in the way that you prefer. It might be worth raising this as an enhancement request to the Jaeger folks - https://github.com/jaegertracing/jaeger

[rineshpk]

@JoshLove-msft Thank you for the clarification. I understood the concept of 'Links' for connecting the spans in the case of batch processing of messages. But, in the below examples from, OTel message spec.

and,

The parent is there. I tried the same example scenarios with the current Azure.Messaging.ServiceBus which is setting the links only not the parent.

[lmolkova]

@rineshpk OTel messaging spec is going through a stabilization process that will change the above examples and the relationship between receive and processing spans. Here are more detals open-telemetry/oteps#192

Note that also the parent is the spec is a different span (one created for 'receive' operation and not 'send').
Unfortunately, the above examples are not feasible to achieve in ServiceBus SDK at this point. We'll wait for the next version of OTel spec.

If you're willing to add a bit of manual instrumentation, you can create a processing span manually and make it a child of producer span (as Microsoft.Azure.ServiceBus did).

You can use ActivitySource.StartActivity overload that takes string parentId and pass content of Diagnostic-Id header as parent.

You'd need your own ActivitySource for it and you can follow OTel docs to configure it.

[rineshpk]

@lmolkova Got it, thank you!