Service with HTTPS endpoint and response compression locks up serving large files

[sergey-m-pega]

Issue Title

Service with HTTPS endpoint and response compression locks up serving large files.

General

Just sign up with Github so that I can report the issue. Luckily I was able to isolate it to the smallest repro solution, which mimics our application setup.

Self hosted test service uses .NET Core v2.1.6 on Windows 10 x64 (and VS 2017 v15.9.1). Test client is a .NET 4.6.1 console application that uses WebClient to communicate with the service.

Service serves files on configured HTTPS endpoint. Response compression is On for HTTP and HTTPS. We respond with "application/octet-stream" content type to the client, same content type we have response compression configured for, which is validated via Postman.

After serving a few large files client locks up. It happens more often than not if files served are large enough, it could be a few 100Kb or over 1Mb. Eventually client errors out with WebException "operation has timed out". Sometimes service shows "Connection id "??????????", Request id "?????????:??????????": the connection was closed because the response was not read by the client at the specified minimum data rate" message after a while, but not all the time.

The odd part is that lock up always happens after serving a few larger files. I could never reproduce it serving small files, no matter how many I tried (ran test against an existing source code base for instance processing .cs files). Additionally, lock ups happen with HTTPS endpoint only. If I switch to HTTP endpoint then it always works fine.

To reproduce the issue, build solution attached. Run service, run console app. Test console application gets a list of all files in "bin" folder and asks for file contents from the service. It then saves the contents to "bin\test" effectively copying the file. You can run test against HTTP or HTTPS endpoint. Hit Enter to proceed with HTTPS endpoint and wait for it to lockup. If it doesn't happen then try the same console app again. Eventually it does lock up.

At this point it's a blocking issues for us. Between choosing HTTPS endpoint or turning response compression off we have to let response compression go, which is a shame. So, I'd appreciate any feedback. Thanks!

ResponseCompressionTest.zip

[karelz]

Does it happen when you change:

1. Server to .NET Framework
2. Client to .NET Core
3. Use HttpClient APIs instead (esp. on .NET Core)

[sergey-m-pega]

Does it happen when you change:

1. Server to .NET Framework
2. Client to .NET Core
3. Use HttpClient APIs instead (esp. on .NET Core)

Our product's client is is locked in to .NET and server to .NET Core so I'm not sure chasing different test conditions is practical in this case. FTR, test case is already attached with this submission.

[karelz]

@sergey-b-berezin chasing where the bug is is however useful for routing to the right team - is it client side or server side?
Is it a new problem or long existing problem?

The "tests" above will answer that.

[sergey-m-pega]

Our client used to use HttpClient but once problems cropped up we switched to "older" WebClient hoping it'd help. It didn't. Everything I tested points at .NET Core server being the issue but I've been wrong before.

Sorry but I don't have the bandwidth to iron out all test permutations you mentioned. That's why I put together an isolated test case, which demonstrates a specific problem, which other tests you suggested may not.

I've got to ask, is this the right project to submit this issue?

[karelz]

If you suspect the server, can you please at least try different client? It should be simple.
Based on that we will route it to the right repo. This repo is useful for nebulous or unclear bug reports.

[sergey-m-pega]

Tried it with HttpClient last night with the same outcome. I built a test client similar to .NET 4.6.1 one included in the test submission, but one built as .NET Core console app instead, with the same logic using HttpClient. So just as I suspected everything points at the server side.

[sergey-m-pega]

What's the next step? Thanks.

[Eilon]

Taking a look now... (Was on vacation last week, sorry for the delay.)

[sergey-m-pega]

Understood. Thanks. It's a delicate balance between annoying someone and asking for help. :)

[Eilon]

OK this is possibly a duplicate of aspnet/BasicMiddleware#247, which we plan to fix in ASP.NET Core 3.0.

If you can get a dump of the ASP.NET Core process while it's hung, then you can open up the dump in VS and see where the thread stacks are. If the stacks look like the ones in aspnet/BasicMiddleware#247, then it is the same issue. If the stacks are different, we will investigate further.

@sergey-m-pega - can you take a look at the threads to see if it's the same issue?

[Eilon]

Tagging @Tratcher here.

[sergey-m-pega]

Will try to collect a dump. Assuming it's the same issue, are there any workarounds exist for this issue in the mean time, short of disabling the response compression entirely? Thanks.

[Eilon]

@Tratcher - what workarounds do you know of?

[Tratcher]

Let's confirm the stack traces before getting deep into the mitigations.

[sergey-m-pega]

Is there post that describes on how to collect it so that it's in the format you expect it to be? Thanks.

[Tratcher]

Using task manager is the easiest, if you have direct access to the machine.
https://blogs.msdn.microsoft.com/debugger/2009/12/30/what-is-a-dump-and-how-do-i-create-one/

[sergey-m-pega]

Got it. Collected 2 dumps, 1 for dotnet.exe and 1 for test client executable via

procdump.exe -ma <XXX.exe> <XXX.dmp>

Let me know if you'd like me to re-run it with different parameters.

Those dump files even zipped are quite large. Where would you suggest posting them for you? Thanks.

[Tratcher]

You can open them yourself in Visual Studio and look at the Threads window to see what callstacks are blocked. Do you see any that resemble those at aspnet/BasicMiddleware#247?

[sergey-m-pega]

I'm not sure how to get that stack trace from dump file tbhwy as I'm not seeing it there. I've trued loading dump file in VS and selecting debug with mixed to see just the regular type stack trace, not even remotely similar to the structure of the above referenced issue.

[Tratcher]

What stack traces do you see in the dotnet.exe dump then?

[sergey-m-pega]

ntdll.dll!00007ffbbe38aa54()
KERNELBASE.dll!00007ffbba8a6099()
[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(-1, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(, )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Id = 25, Status = WaitingForActivation, Method = "{null}", Result = Evaluation of method System.Threading.Tasks.Task`1[System.Threading.Tasks.VoidTaskResult].get
_DebuggerDisplayResultDescription() calls into native method System.Environment.FailFast(). Evaluation of native methods while minidump debugging is not supported.) Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.WebHostExtensions.Run()
ResponseCompressionTest.dll!ResponseCompressionTest.Program.Main({string[1]}) Line 12
at C:\Users???\Documents\Visual Studio 2017\Projects\ResponseCompressionTest\ResponseCompressionTest\Program.cs (12)
[Native to Managed Transition]
hostpolicy.dll!00007ffb90e8fed0()
hostpolicy.dll!00007ffb90e906cd()
hostfxr.dll!00007ffb950be138()
hostfxr.dll!00007ffb950c6e5a()
hostfxr.dll!00007ffb950c4d72()
hostfxr.dll!00007ffb950c3bb6()
hostfxr.dll!00007ffb950bed75()
dotnet.exe!00007ff7d6849b65()
dotnet.exe!00007ff7d684e2c9()
kernel32.dll!00007ffbbe173034()
ntdll.dll!00007ffbbe361471()

[Tratcher]

That's the Main thread, it's expected to be blocked. What about the others?

[sergey-m-pega]

Right, that's why I said earlier that it looks nothing like the other issue stack trace. Dump file handling is something I am not familiar with well enough so I'm not sure how to get the other threads info for you. Please advise. Thanks.

[Tratcher]

It's under Debug -> Windows -> Threads. Look under the Location column to see the stack trace for each threads.
See https://docs.microsoft.com/en-us/visualstudio/debugger/get-started-debugging-multithreaded-apps?view=vs-2017

[sergey-m-pega]

Yeah, I totally forgot that I could look at thread's stack trace there. Never mind... What I was saying earlier was largely related to having dump file only on hand and getting stack trace from that as oppose to attaching to running process.

In any case, I suspect one of ".NET Core ThreadPool" threads is the one you are interested in:

System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait
ntdll.dll!00007ffbbe38aa54()
KERNELBASE.dll!00007ffbba8a6099()
[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(-1, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(, )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Id = 2, Status = WaitingForActivation, Method = "{null}", Result = Evaluation of method System.Threading.Tasks.Task`1[System.Threading.Tasks.VoidTaskResult].get_DebuggerDisplayResultDescription() calls into native method System.Environment.FailFast(). Evaluation of native methods while minidump debugging is not supported.)
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(, , )
System.IO.Compression.dll!System.IO.Compression.DeflateStream.PurgeBuffers()
System.IO.Compression.dll!System.IO.Compression.DeflateStream.Dispose(true)
System.Private.CoreLib.dll!System.IO.Stream.Close()
System.IO.Compression.dll!System.IO.Compression.GZipStream.Dispose(true)
System.Private.CoreLib.dll!System.IO.Stream.Close()
Microsoft.AspNetCore.ResponseCompression.dll!Microsoft.AspNetCore.ResponseCompression.BodyWrapperStream.Dispose()
System.Private.CoreLib.dll!System.IO.Stream.Close()
Microsoft.AspNetCore.ResponseCompression.dll!Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke({Microsoft.AspNetCore.Http.DefaultHttpContext})
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.<Invoke>d__3>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__16>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__18>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__23>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultFilters>d__26>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResultFilterAsync<Microsoft.AspNetCore.Mvc.Filters.IResultFilter, Microsoft.AspNetCore.Mvc.Filters.IAsyncResultFilter>()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResultFilterAsync>d__28<Microsoft.AspNetCore.Mvc.Filters.IResultFilter, Microsoft.AspNetCore.Mvc.Filters.IAsyncResultFilter>>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultAsync>d__20>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Infrastructure.FileResultExecutorBase.WriteFileAsync(, , , )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Mvc.Infrastructure.FileResultExecutorBase.<WriteFileAsync>d__19>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Http.Extensions.dll!Microsoft.AspNetCore.Http.Extensions.StreamCopyOperation.CopyToAsync(, , , , )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Http.Extensions.StreamCopyOperation.<CopyToAsync>d__2>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.ResponseCompression.dll!Microsoft.AspNetCore.ResponseCompression.BodyWrapperStream.WriteAsync(, , , )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.ResponseCompression.BodyWrapperStream.<WriteAsync>d__32>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
System.IO.Compression.dll!System.IO.Compression.DeflateStream.WriteAsyncMemoryCore(, )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.IO.Compression.DeflateStream.<WriteAsyncMemoryCore>d__66>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
System.IO.Compression.dll!System.IO.Compression.DeflateStream.WriteDeflaterOutputAsync()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.IO.Compression.DeflateStream.<WriteDeflaterOutputAsync>d__67>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.FlushAsyncAwaited(, , )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.<FlushAsyncAwaited>d__31>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult()
System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.TrySetResult()
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.OnFlushCompleted()
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.ExecuteWithoutExecutionContext()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.ExecuteWithExecutionContext()
System.IO.Pipelines.dll!System.IO.Pipelines.InlineScheduler.Schedule(, )
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.TrySchedule(, )
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.AdvanceReader(, , , )
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.AdvanceReader(, )
System.IO.Pipelines.dll!System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo()
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.WriteOutputAsync({System.Net.Security.SslStream})
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.<WriteOutputAsync>d__17>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
System.Net.Security.dll!System.Net.Security.SslStreamInternal.WriteAsyncInternal.__ExitWriteAsync|35_0<System.Net.Security.SslStreamInternal.SslWriteAsync>()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.Net.Security.SslStreamInternal.<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d<System.Net.Security.SslStreamInternal.SslWriteAsync>>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
System.Net.Security.dll!System.Net.Security.SslStreamInternal.WriteSingleChunk.__CompleteAsync|36_1<System.Net.Security.SslStreamInternal.SslWriteAsync>(, )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<System.Net.Security.SslStreamInternal.<<WriteSingleChunk>g__CompleteAsync|36_1>d<System.Net.Security.SslStreamInternal.SslWriteAsync>>.MoveNext()
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(, )
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.SetExistingTaskResult()
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult()
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream.WriteAsync(, )
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream.<WriteAsync>d__21>.MoveNext()
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(, , )
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()
[Native to Managed Transition]
kernel32.dll!00007ffbbe173034()
ntdll.dll!00007ffbbe361471()

[Tratcher]

Yes, that's the issue...

System.IO.Compression.dll!System.IO.Compression.DeflateStream.Dispose(true)
System.Private.CoreLib.dll!System.IO.Stream.Close()
System.IO.Compression.dll!System.IO.Compression.GZipStream.Dispose(true)
System.Private.CoreLib.dll!System.IO.Stream.Close()
Microsoft.AspNetCore.ResponseCompression.dll!Microsoft.AspNetCore.ResponseCompression.BodyWrapperStream.Dispose()
System.Private.CoreLib.dll!System.IO.Stream.Close()
Microsoft.AspNetCore.ResponseCompression.dll!Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke({Microsoft.AspNetCore.Http.DefaultHttpContext})

How many threads did you have stuck in a callstack like that? It shouldn't be completely frozen, just slow while it waits for the client to drain data. If you have more threads stuck than the threadpool limit then you might have a problem. Raising the threadpool limit can help mitigate this but not eliminate it.

[sergey-m-pega]

The above stack trace you pointed out looks similar to one from the other issue.

Only one like that that I could see. It gets completely stuck with no CPU on the box and client side eventually timing out. Yeah, I did read about raising thread pool thread limit but if it doesn't fix it permanently we just can't take a chance on it locking up in production.

[sergey-m-pega]

Back to my earlier question please. Are there any workarounds exist til Core 3.0 is released? I assume (future) patch for it is not in 2.2. Thanks.

[Tratcher]

Raising the threadpool limit is the first thing to try.
Disabling compression over HTTPs is the next best option.

[sergey-m-pega]

If raising that limit is guaranteed to fix it then I'd go for it. The last thing we want is to run into occasional production lock ups, which I suspect is something that could still happen being dependent on server's load. So that leaves us with the same option we've already implemented - disabling response compression entirely. That's a far from ideal solution though.

[Tratcher]

The only guaranteed fix requires API and language changes in 3.0 (IAsyncDispose).

Your case is certainly the most severe we've seen, likely because HTTPS compression is off by default and most people use IIS's compression module instead.

[sergey-m-pega]

We'll wait til 3.0 to revisit it then. Thanks for your help.

[karelz]

Closing as there is nothing actionable left here. Let me know if I overlooked something.

[TalAloni]

For anybody still using ASP.NET Core 2.2 and hitting the issue, I want to update that this issue can be worked around by writing another middleware (i.e. ResponseCompressionFlusherMiddleware) that will execute after ResponseCompressionMiddleware and will call FlushAsync on the BodyWrapperStream before ResponseCompressionMiddleware will call Dispose().

The new middleware invoke method should look like this:

public async Task Invoke(HttpContext context)
{
    await _next(context);

    if (context.Response.Headers.ContainsKey(HeaderNames.ContentEncoding)) // We only apply the workaround for compressed responses
    {
        IHttpBufferingFeature bufferingFeature = context.Features.Get<IHttpBufferingFeature>();
        if (bufferingFeature is Stream bodyWrapperStream)
        {
            // Workaround for https://github.com/dotnet/core/issues/2067:
            // When (synchronous) Dispose is called on BodyWrapperStream and not all data had been written, Dispose will invoke (synchronous) Write,
            // which will then (in HttpResponseStream) invoke WriteAsync().GetAwaiter().GetResult(),
            // this logic will cause deadlock due to internal ASP.NET Core 2.2 implementation details.
            // Calling FlushAsync on the BodyWrapperStream will ensure that we do not trigger that condition.
            await bodyWrapperStream.FlushAsync();
        }
    }
}