AddHttpClientInstrumentation creates span under wrong parent when using async/await

[mitchabaza]

Bug Report

List of all OpenTelemetry NuGet
packages and version that you are
using (e.g. OpenTelemetry 1.0.2):

 <package id="OpenTelemetry" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Api" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Api.ProviderBuilderExtensions" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Contrib.Instrumentation.EntityFrameworkCore" version="1.0.0-beta2" targetFramework="net48" />
  <package id="OpenTelemetry.Exporter.Console" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Exporter.InMemory" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Exporter.OpenTelemetryProtocol" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Instrumentation.Http" version="1.7.0" targetFramework="net48" />
  <package id="OpenTelemetry.Instrumentation.SqlClient" version="1.6.0-beta.3" targetFramework="net48" />

Runtime version (e.g. net462, net48, netcoreapp3.1, net6.0 etc. You can
find this information from the *.csproj file):

.NET Framework 4.8

Symptom

AddHttpClientInstrumentation doesn't work when multiple tasks are being executed using async. Namely, HttpClient spans are being associated to the wrong parent.

What is the expected behavior?

All HttpClient calls are attached to the correct parent scope.

What is the actual behavior?

Some HttpClient calls are attached to the wrong (random) parent scope.

Reproduce

I created a simple application that calls Postmans "Echo" API three times, each invocation with a unique query string. Before each HttpClient call, it creates a parent scope called "CallPostManEcho-XX". As such, I would expect to see this trace created

CallPostManEcho-10
-->https://postman-echo.com/get?Id=10
CallPostManEcho-20
-->https://postman-echo.com/get?Id=20
CallPostManEcho-30
-->https://postman-echo.com/get?Id=30

Instead, I see this trace. The http call generated under the CallPostManEcho-10 scope is associated to the CallPostManEcho-30

CallPostManEcho-10
-->
CallPostManEcho-20
-->https://postman-echo.com/get?Id=20
CallPostManEcho-30
-->https://postman-echo.com/get?Id=30
-->https://postman-echo.com/get?Id=10

Failing test cases are available at https://github.com/mitchabaza/OpenTelemetryBug

We will close this issue if:

• The repro project you share with us is complex. We can't investigate custom
  projects, so don't point us to such, please.
• If we can not reproduce the behavior you're reporting.

Additional Context

Add any other context about the problem here.

[martinjt]

I've tried to replicate this but I'm unable to. I assume there is something else in your project, since I copied the WorkerClass into a new .NET Core console app, and added this to Program.cs

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddHttpClientInstrumentation()
                .ConfigureResource(resource => resource.AddService("async-http"))
                .AddSource(nameof(Program))
                .AddOtlpExporter()
                .Build();

var workclass = new WorkerClass();

await workclass.StartAsync();

workclass.StartSync();

tracerProvider.ForceFlush();

Console.WriteLine("done");

This results in 6 traces, each with 2 spans, one of which is the root.

Since this is an older SDK style project format, and there are lots of other things at play in that solution, I can only assume that something else is causing this issue.