System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(bufferLength: 1) failing in CI

[ViktorHofer]

  Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Encoding.Tests (found 338 of 341 test cases)
  Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 4)
    System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(bufferLength: 1) [FAIL]
      Assert.Equal() Failure
                                       ↓ (pos 10246391)
      Expected: ···fghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmn···
      Actual:   ···fghijklmnopqrstuvwxyZabcdefghijklmnopqrstuvwxyzabcdefghijklmn···
                                       ↑ (pos 10246391)
      Stack Trace:
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(630,0): at System.Text.Tests.TranscodingStreamTests.<>c__DisplayClass29_0.<<RunReadTestAsync>g__RunOneTestIteration|3>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(577,0): at System.Text.Tests.TranscodingStreamTests.RunReadTestAsync(Func`4 callback, Boolean suppressExpectedCancellationTokenAsserts)
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(462,0): at System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(Int32 bufferLength)
        --- End of stack trace from previous location ---
  Finished:    System.Text.Encoding.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Encoding.Tests  Total: 11166, Errors: 0, Failed: 1, Skipped: 0, Time: 221.904s
/root/helix/work/workitem/e
----- end Tue Oct 26 10:10:34 UTC 2021 ----- exit code 1 ----------------------------------------------------------

Configuration: net7.0-Linux-Debug-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1440581&view=ms.vss-test-web.build-test-results-tab&runId=41561684&paneView=debug

[ghost]

Tagging subscribers to this area: @tarekgh, @dotnet/area-system-text-encoding
See info in area-owners.md if you want to be subscribed.

Issue Details

---

  Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Encoding.Tests (found 338 of 341 test cases)
  Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 4)
    System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(bufferLength: 1) [FAIL]
      Assert.Equal() Failure
                                       ↓ (pos 10246391)
      Expected: ···fghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmn···
      Actual:   ···fghijklmnopqrstuvwxyZabcdefghijklmnopqrstuvwxyzabcdefghijklmn···
                                       ↑ (pos 10246391)
      Stack Trace:
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(630,0): at System.Text.Tests.TranscodingStreamTests.<>c__DisplayClass29_0.<<RunReadTestAsync>g__RunOneTestIteration|3>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(577,0): at System.Text.Tests.TranscodingStreamTests.RunReadTestAsync(Func`4 callback, Boolean suppressExpectedCancellationTokenAsserts)
        /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs(462,0): at System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(Int32 bufferLength)
        --- End of stack trace from previous location ---
  Finished:    System.Text.Encoding.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Encoding.Tests  Total: 11166, Errors: 0, Failed: 1, Skipped: 0, Time: 221.904s
/root/helix/work/workitem/e
----- end Tue Oct 26 10:10:34 UTC 2021 ----- exit code 1 ----------------------------------------------------------

Configuration: net7.0-Linux-Debug-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1440581&view=ms.vss-test-web.build-test-results-tab&runId=41561684&paneView=debug

Author:	ViktorHofer
Assignees:	-
Labels:	area-System.Text.Encoding, untriaged
Milestone:	-

[stephentoub]

#47444 was tracking the same failure but was closed in August due to no recent occurrences.
cc: @danmoseley, @kunalspathak

[danmoseley]

@MattGal we caught two more fish. (BTW, can we include the machine name in the logs? https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/1440581/logs/2006)

TestResults
| join kind=inner WorkItems on WorkItemId
| join kind=inner Jobs on JobId
| where Finished >= now(-365d)
| where Type == "System.Text.Tests.TranscodingStreamTests"
| where Method == "Read_Span" or Method == "ReadAsync_Memory"
| where Result == 'Fail'
| where QueueName1 == 'ubuntu.1804.armarch.open'
| project Started, Duration, Type, Method, FriendlyName, MachineName, QueueName1, Message, StackTrace, Arguments, Branch

Started	Duration	Type	Method	FriendlyName	MachineName	QueueName1	Message	StackTrace	Arguments	Branch
2021-10-09 07:41:33.3080000	49.227	System.Text.Tests.TranscodingStreamTests	ReadAsync_Memory	System.Text.Encoding.Tests	ddvsotx2l280	ubuntu.1804.armarch.open	Assert.Equal() Failure ↓ (pos 2005207) Expected: ···pqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwx··· Actual: ···pqrstuvwxyzabcdefghiJklmnopqrstuvwxyzabcdefghijklmnopqrstuvwx··· ↑ (pos 2005207)	at System.Text.Tests.TranscodingStreamTests.<>c__DisplayClass29_0.<g__RunOneTestIteration	3>d.MoveNext() in //src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 630 --- End of stack trace from previous location --- at System.Text.Tests.TranscodingStreamTests.RunReadTestAsync(Func`4 callback, Boolean suppressExpectedCancellationTokenAsserts) in //src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 577 at System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(Int32 bufferLength) in /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 462 --- End of stack trace from previous location ---	bufferLength: 1
2021-10-26 10:06:09.4000000	76.945	System.Text.Tests.TranscodingStreamTests	ReadAsync_Memory	System.Text.Encoding.Tests	ddvsotx2l280	ubuntu.1804.armarch.open	Assert.Equal() Failure ↓ (pos 10246391) Expected: ···fghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmn··· Actual: ···fghijklmnopqrstuvwxyZabcdefghijklmnopqrstuvwxyzabcdefghijklmn··· ↑ (pos 10246391)	at System.Text.Tests.TranscodingStreamTests.<>c__DisplayClass29_0.<g__RunOneTestIteration	3>d.MoveNext() in //src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 630 --- End of stack trace from previous location --- at System.Text.Tests.TranscodingStreamTests.RunReadTestAsync(Func`4 callback, Boolean suppressExpectedCancellationTokenAsserts) in //src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 577 at System.Text.Tests.TranscodingStreamTests.ReadAsync_Memory(Int32 bufferLength) in /_/src/libraries/System.Text.Encoding/tests/Encoding/TranscodingStreamTests.cs:line 462 --- End of stack trace from previous location ---	bufferLength: 1

Both single-bit failures, like last time. Both on the SAME machine, ddvsotx2l280, as before.

I believe ddvsotx2l280 has a faulty memory stick and should have its memory replaced. The memory check last time probably just isn't as stressful as all the tests we run hour after hour day after day.

[MattGal]

Pretty cheap to do since the work item already contains it and it's already available from the public workitem API, so filed dotnet/arcade#8112.

I've taken ddvsotx2l280 offline, and I think at this point the thing to do is to just end-of-life it.

[danmoseley]

@MattGal do you need this issue to track disposing of this hapless machine ? if not we can close it

might be worth even burning the name ddvsotx21280 so that there's no confusion in future.

[MattGal]

https://portal.microsofticm.com/imp/v3/incidents/details/269146909/home tracks disposing the hardware, so we don't need this issue to remain open to prevent it in the future; close as you like. I also verified with DDFUN that the machine has been manually disabled so cannot accidentally come back online.

My understanding is that newer machines would just be +1 to the max previous value and/or a totally new naming strategy based off the hardware type (that is, I doubt we'll acquire more TX2s), so I don't think the name needs help to be retired.