Repro hang of integration tests after upgrading to ASP.NET Core 6

[angularsen]

Problem description

After upgrading from ASP.NET 5 to 6, our integration tests would hang indefinitely on Azure Pipelines with ubuntu-latest.

After analyzing dumps with dotnet test --blame-hang-timeout, we found a deadlock on WebApplicationFactory.Dispose() and concurrent tests calling the API.

The root cause

• Single WebApplicationFactory instance per test class, reused by its test methods to call the API.
• Hosted service with Task.Delay(TimeSpan.FromSeconds(5), ct) in its IHostedService.StopAsync(), which is used to flush logs and telemetry before the API is allowed to shut down.
• The delay seems to create a deadlock for shared factories, only on Linux/Ubuntu and only on .NET6.

Workaround

Do not delay IHostedService.StopAsync() for "Test" environment.

Investigation

• dotnet test on Azure Pipelines ubuntu-20.04 would hang indefinitely after upgrading to ASP.NET Core 6
• Could not reproduce on Azure Pipelines windows-2019
• Could not reproduce locally with Rider or Visual Studio 2022 on Windows
• Could not reproduce with dotnet test on Windows
• Could not reproduce on .NET5
• Could not reproduce on .NET6 when creating individual factories per test method

Test setup

This mimics our particular test setup:

• xunit tests
• AddHostedService<MySlowStoppingHostedService>(), to add a 5 second delay on StopAsync().
• Multiple integration test classes, using IClassFixture<TWebApplicationFactory> to create a shared factory for each class, reused by its test methods that call one or more API methods.

Repro

Clone agl/test-repro branch.

Run tests:

cd src
dotnet test Net6WebApi.sln --logger "console;verbosity=detailed"

Test setup

Test.Net5WebApi are ASP.NET Core 5 integration tests (all green).
Test.Net6WebApi are ASP.NET Core 6 integration tests (red and green).

Common test setup for both:

• Repro_GREEN_ConcurrentTestsWithIndividualFactories
• Repro_RED_ConcurrentTestsOnSharedFactory

On Ubuntu 20.04 - dotnet test hangs

Observed on both ubuntu-20.04 in Azure Pipelines and WSL Ubuntu 20.10 locally, with .NET 6.0.100 SDK.

On Windows - Test cleanup fails, but test does not hang

Windows 10 with .NET 6.0.100 SDK.

[Test Class Cleanup Failure (Test.Net6WebApi.Repro_RED_ConcurrentTestsOnSharedFactory+TestClass3)]: System.AggregateException : One or more hosted services failed to stop. (A task was canceled.)
---- System.Threading.Tasks.TaskCanceledException : A task was canceled.
  Stack Trace:
     at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DisposeAsync()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose(Boolean disposing)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose()
----- Inner Stack Trace -----
   at Test.Net6WebApi.Setup.MySlowStoppingHostedService.StopAsync(CancellationToken cancellationToken) in C:\dev\digma\NET6-Testing\src\Test.Net6WebApi\Setup\MySlowStoppingHostedService.cs:line 30
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)