ExecutionContext_NotCachedInSocketAsyncEventArgs - failure in Assert.True

[trylek]

Platform: Linux_musl x64 Release
Pipeline: runtime-libraries outerloop
Example run: https://dev.azure.com/dnceng/public/_build/results?buildId=482386&view=logs&j=9edfae9b-c909-5390-dc2d-2665643e0586
Proximate diagnostic info:

  Starting:    System.Net.Sockets.Tests (parallel test collections = on, max threads = 2)
    System.Net.Sockets.Tests.ExecutionContextFlowTest.ExecutionContext_NotCachedInSocketAsyncEventArgs [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/ExecutionContextFlowTest.cs(535,0): at System.Net.Sockets.Tests.ExecutionContextFlowTest.ExecutionContext_NotCachedInSocketAsyncEventArgs()
  Finished:    System.Net.Sockets.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Sockets.Tests  Total: 515, Errors: 0, Failed: 1, Skipped: 0, Time: 91.496s

[BruceForstall]

@karelz You marked this as "Future", but it's failing in the CI. Are you going to disable the test so we don't see this noise?

[karelz]

@BruceForstall what kind of noise do you see from this? We are actively monitoring test results - mainly official runs and address things affecting CI. This one did not pop for us yet.
Note: Networking tests are inherently difficult to make 100% passing - our current bar is 1 failure per week to investigate.

cc @alnikola

[BruceForstall]

@karelz This particular test I see has failed twice since May. Overall, it looks like Sysetm.Net has 28 failures in the last 30 days (after I ignored PostAsync_Cancel_CancellationTokenPassedToContent which had hundreds of failures).

Maybe we need a different model for running/monitoring these tests, to avoid the noise. E.g., maybe System.Net tests should run in their own Azure DevOps pipelines, separate from the rest of the tests, and our vendor teams that investigate CI system failures and ignore them, and your team can be responsible for monitoring. We can discuss offline if you wish.

[alnikola]

@BruceForstall I checked this test history from 1st February till 18th June and found only 2 failures in 2 different months which doesn't look disruptive and is far below our threshold.

Method	Started	Message
ExecutionContext_NotCachedInSocketAsyncEventArgs	2020-06-15 11:49:58.1910000	Assert.True() Failure\nExpected: True\nActual: False
ExecutionContext_NotCachedInSocketAsyncEventArgs	2020-05-03 12:07:29.5040000	Assert.True() Failure\r\nExpected: True\r\nActual: False

[karelz]

Failures during 3/12-9/6 (incl. PRs):

Date	Build / PR	OS
4/6	Official run	Fedora.32.Amd64.Open
4/9	Official run	Fedora.30.Amd64.Open
4/23	Official run	Debian.10.Amd64.Open
5/19	Official run	SLES.15.Amd64.Open - Mono
5/19	Official run	Centos.8.Amd64.Open - Mono
5/20 - 6/2	PR #53027 (wasm)	26x failures - Mono
5/26	Official run	Fedora.32.Amd64.Open - Mono
5/26	Official run	Centos.8.Amd64.Open - Mono
5/26	Official run	Ubuntu.1910.Amd64.Open - Mono
5/26	PR #53277	Debian.10.Amd64.Open - Mono
5/26 - 5/27	PR #53280	5x failures - Mono
6/9	Official run	Fedora.32.Amd64.Open
6/9	PR #52923	Debian.10.Amd64.Open
6/17	PR #54289	3x failures - Mono
6/23 - 6/24	PR #54618	4x failures - Mono
6/25		... includes release/6.0 branches from now on ...
6/27 @ 12:44:57	Official run - 6.0-preview6	Fedora.30.Amd64.Open
6/27 @ 12:46:49	Official run	Ubuntu.1604.Amd64.Open
7/3	Official run - 6.0-preview3	Ubuntu.1604.Amd64.Open
7/8	Official run - 6.0-preview4	SLES.15.Amd64.Open
7/11	Official run	OSX.1013.Amd64.Open
7/17	Official run - 6.0-preview6	Fedora.32.Amd64.Open
7/19	Official run - 6.0-preview3	SLES.15.Amd64.Open
7/24	Official run - 6.0-preview6	SLES.12.Amd64.Open
7/25 @ 12:01:58	Official run - 6.0-preview6	OSX.1015.Amd64.Open
7/25 @ 12:19:23	Official run - 6.0-preview7	SLES.15.Amd64.Open
7/27	Official run - 6.0-preview7	Windows.81.Amd64.Open
7/30 @ 12:23:23	Official run - 6.0-preview4	SLES.15.Amd64.Open
7/30 @ 19:13:40	PR #56629 (off main)	Centos.8.Amd64.Open - Mono
7/30 @ 19:19:21	PR #56629 (off main)	Ubuntu.1910.Amd64.Open - Mono
7/31	Official run - 6.0-preview6	Alpine.312.Amd64.Open
8/2 @ 12:29:58	Official run - 6.0-preview5	Fedora.32.Amd64.Open
8/2 @ 12:35:31	Official run - 6.0-preview3	Fedora.32.Amd64.Open
8/3	Official run - 6.0-preview5	Ubuntu.1804.Amd64.Open
8/8	Official run - 6.0-preview7	Ubuntu.1604.Amd64.Open
8/11	Official run - 6.0-preview7	SLES.12.Amd64.Open
8/16	PR #57503 (off main)	OSX.1015.Amd64.Open
8/25	PR #57745 (off main)	RedHat.7.Amd64.Open - Mono
8/27	Official run - 6.0-rc1	Alpine.312.Amd64.Open
9/3	Official run - 6.0	Alpine.312.Amd64.Open
9/5	Official run	Fedora.34.Amd64.Open
9/6	Official run	Windows.81.Amd64.Open

[karelz]

@antonfirsov another Sockets test failure that is happening more on release/6.0 branches ...

[antonfirsov]

No recent failures, closing.