HttpClient Response Stream hangs and doesn't respect timeout (only HttpCompletionOption.ResponseHeadersRead) #36822
Comments
Dotnet-GitSync-Bot
added
area-System.Net.Http
untriaged
|
Tagging subscribers to this area: @dotnet/ncl |
prcdpr
changed the title
Can you be more specific about exactly what you're doing here? Code would be helpful.
Are you passing the cancellation token to CopyToAsync? |
Which cancellation token ( HttpResponseMessage response = await client.GetAsync(uri, HttpCompletionOptions.ResponseHeadersRead, cts.Token);That cancellation token only affects the initial call to the GetAsync() API. Once it returns, that cancellation token isn't used anymore for any further I/O on the HTTP response stream. And since you are using ResponseHeadersRead, the GetAsync() API should be returning quickly after just getting the response header.
Are you passing a cancellation token to the CopyToAsync() API? Can you show the C# code that you're using to call this API? |
|
This is how I set timeout I pass cancelation token only to SendAsync, I didn't pass it to CopyToAsync. But I would expect when underlying network socket breaks due to poor internet, the Response Stream would be automatically closed as there won't be any additional bytes to read from this socket. |
marklio
added
the
tenet-reliability
|
I could pass cancelation token to CopyToAsync method but consider this example.
|
|
The read in CopyToAsync will wake up when the OS notifies it that the connection has closed. It's quite likely the OS simply hasn't recognized that the connection dropped. TCP is quite resilient. |
|
(And if you want your cancellation token to cancel all reading from the response stream, you should pass it to CopyToAsync.) |
|
I had a case of my user waiting for more than 1 day in such frozen state for CopyToAsync to finish. It's quite possible what you say that that OS didn't recognize the connection drop. In that case it wouldn't qualify as bug but to me it's still a gotcha that can be quite costly and maybe it should be documented how you should work with Response Stream when you set HttpCompletionOption.ResponseHeadersRead flag? Especially that HttpClient user should be aware that Response Stream may become stuck and it's user responsibility to handle reading timeout once again. |
|
@prcdpr can you please supply some code demonstrating the issue? |
|
OK so there is the repro I managed to pull off. After around 10s after application start, I manually disconnect LAN cable between switch and router. using System;
using System.Globalization;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
static async Task Main(string[] args)
{
var httpClient = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, "https://speed.hetzner.de/10GB.bin");
var response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
Console.WriteLine($"Content-Length: {response.Content.Headers.ContentLength.Value}");
var responseStream = await response.Content.ReadAsStreamAsync();
var memoryStream = new MemoryStream();
_ = Task.Run(async () =>
{
while (true)
{
Console.WriteLine($"[{DateTimeOffset.Now.ToString(CultureInfo.InvariantCulture)}] Memory stream length: {memoryStream.Length}");
await Task.Delay(1000);
}
});
await responseStream.CopyToAsync(memoryStream);
}
}
}However, if I disconnect cable between my desktop computer and switch, it stops copying streams but as soon as I reconnect the cable it doesn't get stuck and it actually resumes download. |
|
I wonder how is it related to following properties of response Stream
It looks like setting ReadTimeout on the response stream is unsupported, it throws an exception when I try to do that. |
|
TCP connection stays is in ESTABLISHED state even if network cable is disconnected for more than 10 minutes I'm no expert on TCP stack but shouldn't there be some kind of timeout on the socket?
|
|
That depends if you have data flowing or not. In the former case, TCP should notice lack of ACK and abort. For idle connections, one can enable TCP Keepalive but that is not easily accessible with current HTTP API. On some OSes, that can be enabled globally via networking defaults. |
|
So out of curiosity I tried to use this method on the beginning of my code With timeout set to 5000ms and interval set to and 5000ms, but it doesn't unstuck the copying. Is this setting effective to HTTP API? |
No. ServicePointManager/ServicePoint are legacy objects that are from .NET Framework. They have no affect in .NET Core. |
|
ServicePoint is not relevant in .NET Core. That is legacy .NET Framework API. The docs should make it more clear ;( |
|
Well, so maybe my issue should qualify not as a bug but as a feature request to allow developers setting TCP KeepAlive in System.Net.Http.HttpClient? There are hardly any alternatives to HttpClient other than using libcurl bindings or manually working with sockets so it may become a blocker for many people working with large streaming requests. |
|
That work is coming in in 5.0 @prcdpr. If you don't want overall timeout but more timeout between reads, do not use CopyToAsync() but implement similar function with timer between reads. That would allow you to be more nimble on network failures. |
|
I'll do it that way then. Do you already have it tracked for 5.0 so I can close this issue and subscribe to the proper one? |
|
Thanks for help |
ghost
locked as resolved and limited conversation to collaborators
Description
This is a bug that caused me to waste many hours debugging and is very hard to reproduce.
There are requirements for this bug to happen:
Sometimes, usually on poor internet connection it happens that Response stream is never closed even though cancelation token is set.
Therefore it causes CopyToAsync to hang forever.
I have found workaround in this article: https://www.danielcrabtree.com/blog/33/gotchas-with-httpclients-cancelpendingrequests-and-timeout-in-net
Workaround is pretty simple, you need to create another cancelation token and close response stream forcibly after timeout.
Configuration
Regression?
Don't know.
Other information
In my company I had at least dozen user reports usually from remote workers with poor internet connection that the company's downloader app freezes and stops downloading.
Myself I have never experienced it which is probably because I have fast fiber internet at home.
The text was updated successfully, but these errors were encountered: