Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled #40436

Closed
GrabYourPitchforks opened this issue Aug 6, 2020 · 4 comments · Fixed by #40464 or #40844
Closed

Flaky test: Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled #40436

GrabYourPitchforks opened this issue Aug 6, 2020 · 4 comments · Fixed by #40464 or #40844
Assignees
@alnikola
Labels
area-System.Net.Http blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' test-bug Problem in test source code (most likely)
Milestone
5.0.0

Comments

@GrabYourPitchforks
Copy link
Member

Test Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled appears to be flaky. Occasionally timing out and causing CI failures.

See:
https://github.com/dotnet/runtime/pull/36252/checks?check_run_id=951684309
https://github.com/dotnet/runtime/pull/40426/checks?check_run_id=951565333

    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled [FAIL]
      System.TimeoutException : Task timed out after 00:01:00
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(41,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(2213,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.PrepareConnection(Http2LoopbackServer server, HttpClient client, UInt32 maxConcurrentStreams, Int32 readTimeout, Int32 expectedWarpUpTasks)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(2008,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled()
        --- End of stack trace from previous location ---
  Finished:    System.Net.Http.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 2085, Errors: 0, Failed: 1, Skipped: 14, Time: 131.149s
@GrabYourPitchforks GrabYourPitchforks added area-System.Net.Http test-bug Problem in test source code (most likely) blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels Aug 6, 2020
@ghost
Copy link

ghost commented Aug 6, 2020

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Aug 6, 2020
@alnikola alnikola mentioned this issue Aug 6, 2020
Merged
@jashook jashook mentioned this issue Aug 6, 2020
Closed
@scalablecory scalablecory added this to the 5.0.0 milestone Aug 6, 2020
@scalablecory scalablecory removed the untriaged New issue has not been triaged by the area owner label Aug 6, 2020
alnikola added a commit that referenced this issue Aug 7, 2020
@alnikola
Increase multiple HTTP/2 connections test timeouts again (#40464)
5ca480e 
Fixes #40115
Fixes #40436
Jacksondr5 pushed a commit to Jacksondr5/runtime that referenced this issue Aug 10, 2020
@alnikola @Jacksondr5
Increase multiple HTTP/2 connections test timeouts again (dotnet#40464)
e1fde00 
Fixes dotnet#40115
Fixes dotnet#40436
@MihaZupan
Copy link
Member

Looks like the test is failing even after #40464

@MihaZupan MihaZupan reopened this Aug 11, 2020
@danmoseley
Copy link
Member

Failed again on net5.0-Linux-Debug-x64-CoreCLR_release-Centos.7.Amd64.Open

ystem.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.\n---- System.TimeoutException : The operation was canceled.\n-------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.


Stack trace
   at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 595
   at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 37
   at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.PrepareConnection(Http2LoopbackServer server, HttpClient client, UInt32 maxConcurrentStreams, Int32 readTimeout, Int32 expectedWarpUpTasks) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 2214
   at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 2009
--- End of stack trace from previous location ---
----- Inner Stack Trace -----

----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__67_0(Object s) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1216
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 694
   at System.Threading.CancellationTokenSource.<>c.<.cctor>b__56_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 35
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 627
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 589
   at System.Threading.TimerQueue.FireNextTimers() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 274
--- End of stack trace from previous location ---
   at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1176
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 849
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1806
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 871
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.FinishSendAsync(ValueTask`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, Boolean buffered, Boolean async, CancellationToken callerToken, Int64 timeoutTime) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 565

https://dev.azure.com/dnceng/public/_build/results?buildId=767524&view=ms.vss-test-web.build-test-results-tab&runId=23867748&paneView=debug&resultId=178458

Again it's Libraries Test Run release coreclr Linux x64 Debug

hit in #40729

@antonfirsov
Copy link
Member

Hit in #40506

https://dev.azure.com/dnceng/public/_build/results?buildId=767472&view=ms.vss-test-web.build-test-results-tab&runId=23866186&resultId=178920&paneView=debug

@antonfirsov antonfirsov mentioned this issue Aug 14, 2020
Merged
@alnikola alnikola mentioned this issue Aug 14, 2020
Merged
alnikola added a commit that referenced this issue Aug 14, 2020
@alnikola
Flaky multiple HTTP/2 connection tests read full request body (#40844)
056f728 
There are 2 flaky HTTP/2 tests verifying multiple connections feature which are randomly failing on CI, but not locally.
- Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled
- Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated

It seems the failure is caused by not reading the request body. In current implementation, `Http2LoopbackServer` read only HEADERS frame via `ReadRequestHeaderAsync` and then immediately sends response. However, the client firstly completely sends headers and body and only then starting reading a response. Thus, it seems to get blocked sometimes if the server didn't read the full body. This PR fixes this by calling `ReadAndParseRequestHeaderAsync` instead of `ReadRequestHeaderAsync`.

Fixes #40436
Fixes #40115
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@alnikola alnikola

Labels
area-System.Net.Http blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' test-bug Problem in test source code (most likely)
Projects
None yet
Milestone
5.0.0
7 participants
@GrabYourPitchforks @scalablecory @danmoseley @antonfirsov @MihaZupan @Dotnet-GitSync-Bot @alnikola