[QUERY] Application Insights dependency auto-collection support for Azure.Messaging.ServiceBus

[DSTAlan]

Query/Question
I am using Azure.Messaging.ServiceBus with a new Azure Function integrated with Application Insights. Should the Service Bus be automatically detected as a dependency by Application Insights and show up in the Application Map? It isn't listed as supported at Dependency auto-collection so I'm assuming not? If this is the case, is there a plan for this to be implemented in the future or documentation on how to manually set up the dependency collection with the current version?

Environment:

• Azure.Messaging.ServiceBus 7.0.1
• Microsoft.Azure.WebJobs.Logging.ApplicationInsights 3.0.23
• Currently running locally on Windows 10 with .Net Core 3.1
• Visual Studio 16.8.3

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

@pakrym - If you have any thoughts here, would you be so kind as to share them?

[alanrroberts]

Just to add that I've found removing Microsoft.ApplicationInsights.WorkerService 2.16.0 from my project allowed the dependency auto-collection support to start working with Azure.Messaging.ServiceBus 7.0.1 so there seems to be an incompatibility of some kind there.

[ramya-rao-a]

@JoshLove-msft @pakrym Any updates here? If this is a valid feature request or a bug, can you please add the appropriate label and milestone?

[shlomiassaf]

Basically, everything is already done in Azure.Messaging.ServiceBus.Diagnostics.

The only thing left to do is to add a dependency tracker to add additional data.

In our setup, using Microsoft.ApplicationInsights.WorkerService we do get events posted to insights as "Request" but the operation name is "ServiceBusProcessor.ProcessMessage" and the target/source is not added as a metric so we can't really figure out what is the entity in question. We guess it based on the operations in the transaction :)

Also worth noting that admin operations (e.g. create topic) are also added, their type is properly identified as "Azure Service Bus" which is because they are HTTP operations (XML API) which the library knows to identify as azure service bus operation. I guess this is left over from the old library where it had the same admin api.

I believe some minor tweaks are required to fully support this library, as the AzureSdkDiagnosticsEventHandler does most of the job but it's too generic to do it 100%.

[JoshLove-msft]

In our setup, using Microsoft.ApplicationInsights.WorkerService we do get events posted to insights as "Request" but the operation name is "ServiceBusProcessor.ProcessMessage" and the target/source is not added as a metric so we can't really figure out what is the entity in question. We guess it based on the operations in the transaction :)

We are adding an attribute that has the entity name - https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/src/Diagnostics/DiagnosticProperty.cs#L19

Are you not seeing this?

[JoshLove-msft]

Possibly related - microsoft/ApplicationInsights-dotnet#2151

[shlomiassaf]

@JoshLove-msft While it's in the Activity, along with other data you add, it is not used when transforming it to an ITelemetry for application insights.

[JoshLove-msft]

@JoshLove-msft While it's in the Activity, along with other data you add, it is not used when transforming it to an ITelemetry for application insights.

@pakrym @lmolkova do you have any insight here into what we are missing from the SDK side?

[lmolkova]

Yes, ApplicationInsights SDK does not support collecting Track2 ServiceBus signals and needs a bit of work. It supports Track1 SDK though. I'm planning it for the next semester (first half of 2022).

[shlomiassaf]

@lmolkova The library is great, but maybe providing some hooking points to certain modules will allow users to opt-in before official support it available. The 7.x versions are here for quite long.

It will also allow users to add additional data, where need, instead of writing custom events.

[lmolkova]

@shlomiassaf unfortunately there are no preview bits you could try. As you mentioned, we'd need something very similar to AzureSdkDiagnosticsEventHandler that takes care of creating requests/dependencies and setting all properties similar to EventHubs.

If by any means you could contribute it to the Application Insights codebase earlier than next year, I can review it and work with the Application Insights team to get it merged.

[shlomiassaf]

For anyone who tackles this, here's a quick workaround until this is fixed:

Create (and register) an ITelemetryInitializer which is used to propagate service bus information into the telemetry.

public class ServiceBusTelemetryInitializer : ITelemetryInitializer
    {
        public const string AzureServiceBus = "Azure Service Bus";

        public ServiceBusTelemetryInitializer()
        {
        }
        
        public void Initialize(ITelemetry telemetry)
        {
            try
            {
                if (Activity.Current != null) // Is this possible in an initializer???
                {
                    if (telemetry is RequestTelemetry req &&
                        (req.Name == "ServiceBusProcessor.ProcessMessage" || req.Name == "ServiceBusSessionProcessor.ProcessSessionMessage"))
                    {
                        var entity = Activity.Current.Tags
                            .FirstOrDefault(t => t.Key == "message_bus.destination").Value;
                        var endpoint = Activity.Current.Tags
                            .FirstOrDefault(t => t.Key == "peer.address").Value;

                        req.Properties["Type"] = AzureServiceBus;
                        req.Context.Operation.Name = entity;
                        req.Source = string
                            .Format(CultureInfo.InvariantCulture, "type:{0} | entity:{1} | endpoint:{2}", AzureServiceBus, entity, endpoint);

                    }
                    else if (telemetry is DependencyTelemetry {Name: "ServiceBusSender.Send"} send)
                    {
                        var entity = Activity.Current.Tags
                            .FirstOrDefault(t => t.Key == "message_bus.destination").Value;
                        var endpoint = Activity.Current.Tags
                            .FirstOrDefault(t => t.Key == "peer.address").Value;

                        send.Type = AzureServiceBus;
                        send.Context.Operation.Name = entity;
                        send.Target = string.Join(" | ", endpoint, entity);
                    }   
                }
            }
            catch
            {
                // ignore
            }
        }

    }

The only issue is duplicated events

On each send there are 2 events:

• ServiceBusSender.Send
• Message

For each processor received message there are 2 events:

• ServiceBusReceiver.Receive
• ServiceBusProcessor.ProcessMessage or ServiceBusSessionProcessor.ProcessSessionMessage

The "Receive" is not only when receiving, it's every polling timeout as well, I see a lot of 1 minute hits.
I did not try receiving directly, as I use processor but I wonder how one will do it?
It will keep bombarding tracking of polling of receive which only end without a hit.

For that you can use a processor:

 public class ServiceBusTelemetryProcessor : ITelemetryProcessor
    {
        private ITelemetryProcessor Next { get; }

        // next will point to the next TelemetryProcessor in the chain.
        public ServiceBusTelemetryProcessor(ITelemetryProcessor next) => Next = next;
        
        public void Process(ITelemetry item)
        {
            if (item is RequestTelemetry {Name: "ServiceBusReceiver.Receive"})
                return;
            if (item is DependencyTelemetry {Name: "Message"})
                return;
            
            Next.Process(item);
        }
        
    }

[Shishkovskiy]

@lmolkova Hi there, any updates ? 🙂

[bh3605]

Well, this explains why I have one service showing these 3 new dependencies. Thanks!

We're using NServicebus and I've already wired up full tracing. Is there something about these dependencies that I could leverage instead, or should I write another filter to ignore these 3? ServiceBusReceiver.Receive, Message, and ServiceBusSender.Send

[JoshLove-msft]

The AppInsights update is available in https://www.nuget.org/packages/Microsoft.ApplicationInsights/2.21.0. The Functions integration will light up once Azure/azure-webjobs-sdk#2900 is addressed.