Not all ASP.NET requests are exported

[pardahlman]

Bug Report

List of NuGet packages and
version:

• OpenTelemetry.Instrumentation.AspNet 1.0.0-rc1.1
• OpenTelemetry.Instrumentation.Http 1.0.0-rc1.1
• OpenTelemetry.Contrib.Exporter.Stackdriver local build

Runtime version:

• net472

Symptom

We are running an ASP.NET (classic) application with the telemetry module provided by Microsoft in Microsoft.AspNet.TelemetryCorrelation 1.0.8. HTTP requests and ASP.NET instrumentation are enabled and exported to Stackdriver through a local build of the contrib project with updated dependecy to underlying open-telemetry package to 1.0.0-rc1.1. For many requests, this works as expected and trace data is ingested by Cloud Trace. However, some requests are never properly instrumented. One may suspect the local build of Stackdriver as the culprit, but I believe that the issue lays somewhere in the ASP.NET instrumentation library. After debugging the code I have made the following observations:

When everything works:

• The activity Microsoft.AspNet.HttpReqIn is correctly created by the TelemetryCorrelationHttpModule.
• The method OnStartActivity on OpenTelemetry.Instrumentation.AspNet.Implementation.HttpInListener is called
• The AspNet-activity is stopped
• The method OnStopActivity on OpenTelemetry.Instrumentation.AspNet.Implementation.HttpInListener is called
• Export is triggered on registered exporters

When traces are not exported:

• The activity Microsoft.AspNet.HttpReqIn is correctly created by the TelemetryCorrelationHttpModule.
• The method OnStartActivity on OpenTelemetry.Instrumentation.AspNet.Implementation.HttpInListener is called
• The AspNet-activity is stopped
• The method OnStopActivity on OpenTelemetry.Instrumentation.AspNet.Implementation.HttpInListener is not called
• Export is not triggered on registered exporters

When backtracking why OnStopActivity is not called, I noticed that OpenTelemetry.Instrumentation.DiagnosticSourceListener's OnNext method bails out here when called with Microsoft.AspNet.HttpReqIn.Stop kvp because Activity.Current is null.

What is the expected behavior?

All HTTP requests to the ASP.NET should be instrumented and exported

What is the actual behavior?

Some http requests are not instrumented and exported

Reproduce

Unfortunately, I have not been able to create a small repro app. I will continue to investigate this issue and see if I can get back with easier repro-steps.

[akselsson]

Do note that we noticed this behaviour on a ServiceStack application, running the last BSD-licensed version of ServiceStack.

Here's an attempt to reproduce the problem:
https://github.com/akselsson/OpenTelemetryServiceStackInvestigation

EDIT: I updated the example with example of what happens if a servicestack handler contains code that starts an activity (e.g. a http client call)

[cijothomas]

If the activity.current is null during OnEnd callback, then it could be a bug outside of this repo. (Like AspNetTelemetryCorrelation module.)

Is this only occurring with ServiceStack?

[akselsson]

Is this only occurring with ServiceStack?

As far as we know, yes. The HttpReqIn activity is finished in another plain asp.net mvc site that we host.

[tijlreynhout]

We are having the same problem when forwarding a http request from one .net wepabi (Server A) to another .net webapi (Server B). When inspecting the Activity.Current in an Owin middleware on Server B we also see "Microsoft.AspNet.HttpReqIn". Opentelemtry is configured for Server B but the trace for this request is not exported. If we call Server B directly, tracing works as expected. What is the status of this ticket?

[nipunsahnieptura]

We are using OTEL to enable monitoring for our .NET Application and are experiencing the same issue where the traces are generated with limited information and not all activities are recorded.

I suspect, the issue is similar to what has been described here, is there any update on this issue ?