Microsoft.CodeAnalysis.Testing.AnalyzerTest.RunAsync() fails when built on Windows with Linux line endings

[oguzeldereli1]

When the dotnet/runtime repo is copied to a Windows machine from a Linux setup, LibraryImportGenerator.Unit.Tests fail (along with several other libraries dotnet/runtime#66600).

<test name="LibraryImportGenerator.UnitTests.CustomTypeMarshallerFixerTests.CustomTypeMarshallerWithTwoStageMarshallingMethod_NoFeature_ReportsDiagnostic" type="LibraryImportGenerator.UnitTests.CustomTypeMarshallerFixerTests" method="CustomTypeMarshallerWithTwoStageMarshallingMethod_NoFeature_ReportsDiagnostic" time="0.3910885" result="Fail">
        <failure exception-type="Xunit.Sdk.TrueException">
          <message><![CDATA[Context: Iterative code fix application\r\ncontent of '/0/Test0.cs' did not match. Diff shown with expected as baseline:\r\n <LF>\r\n using System;<LF>\r\n using System.Runtime.InteropServices;<LF>\r\n <LF>struct S { }<LF>\r\n <LF>[CustomTypeMarshaller(typeof(S), CustomTypeMarshallerKind.Value, Features = CustomTypeMarshallerFeatures.TwoStageMarshalling)]<LF>\r\n unsafe struct Native<LF>\r\n {<LF>\r\n     public Native(S s){}<LF>\r\n <LF>    public int ToNativeValue() => throw null;<LF>\r\n-<LF>    public S ToManaged() => throw null;<LF>\r\n-<LF>    public void FromNativeValue(int value)<LF>\r\n-    {<LF>\r\n-        throw new NotImplementedException();<LF>\r\n-    }<LF>\r\n+<LF>    public S ToManaged() => throw null;<CR><LF>\r\n+<CR><LF>\r\n+    public void FromNativeValue(int value)<CR><LF>\r\n+    {<CR><LF>\r\n+        throw new NotImplementedException();<CR><LF>\r\n+    }<CR><LF>\r\n }<LF>\r\n <LF>[CustomTypeMarshaller(typeof(S), CustomTypeMarshallerKind.Value, Features = CustomTypeMarshallerFeatures.TwoStageMarshalling)]<LF>\r\n unsafe struct Native2<LF>\r\n {<LF>\r\n     public Native2(S s){}<LF>\r\n <LF>    public void FromNativeValue(int value) { }<LF>\r\n-<LF>    public S ToManaged() => throw null;<LF>\r\n-<LF>    public int ToNativeValue()<LF>\r\n-    {<LF>\r\n-        throw new NotImplementedException();<LF>\r\n-    }<LF>\r\n+<LF>    public S ToManaged() => throw null;<CR><LF>\r\n+<CR><LF>\r\n+    public int ToNativeValue()<CR><LF>\r\n+    {<CR><LF>\r\n+        throw new NotImplementedException();<CR><LF>\r\n+    }<CR><LF>\r\n }\r\n\r\nExpected: True\r\nActual:   False]]></message>
          <stack-trace><![CDATA[   at Microsoft.CodeAnalysis.Testing.Verifiers.XUnitVerifier.Fail(String message) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Testing.Verifiers.XUnit/XUnitVerifier.cs:line 87
   at Microsoft.CodeAnalysis.Testing.IVerifierExtensions.EqualOrDiff(IVerifier verifier, String expected, String actual, String message) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing/Extensions/IVerifierExtensions.cs:line 67
   at Microsoft.CodeAnalysis.Testing.CodeFixTest`1.VerifyProjectAsync(ProjectState newState, Project project, IVerifier verifier, CancellationToken cancellationToken) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs:line 528
   at Microsoft.CodeAnalysis.Testing.CodeFixTest`1.VerifyFixAsync(String language, ImmutableArray`1 analyzers, ImmutableArray`1 codeFixProviders, SolutionState oldState, SolutionState newState, Int32 numberOfIterations, Func`10 getFixedProject, IVerifier verifier, CancellationToken cancellationToken) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs:line 506
   at Microsoft.CodeAnalysis.Testing.CodeFixTest`1.VerifyFixAsync(SolutionState testState, SolutionState fixedState, SolutionState batchFixedState, IVerifier verifier, CancellationToken cancellationToken) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs:line 470
   at Microsoft.CodeAnalysis.Testing.CodeFixTest`1.RunImplAsync(CancellationToken cancellationToken) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.CodeFix.Testing/CodeFixTest`1.cs:line 309
   at Microsoft.CodeAnalysis.Testing.AnalyzerTest`1.RunAsync(CancellationToken cancellationToken) in /_/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing/AnalyzerTest`1.cs:line 182
   at LibraryImportGenerator.UnitTests.Verifiers.CSharpCodeFixVerifier`2.VerifyCodeFixAsync(String source, DiagnosticResult[] expected, String fixedSource, Int32 numIncrementalIterations, Int32 numFixAllIterations) in D:\Runtime\runtime\src\libraries\System.Runtime.InteropServices\tests\LibraryImportGenerator.UnitTests\Verifiers\CSharpCodeFixVerifier.cs:line 94
   at LibraryImportGenerator.UnitTests.CustomTypeMarshallerFixerTests.CustomTypeMarshallerWithTwoStageMarshallingMethod_NoFeature_ReportsDiagnostic() in D:\Runtime\runtime\src\libraries\System.Runtime.InteropServices\tests\LibraryImportGenerator.UnitTests\CustomTypeMarshallerFixerTests.cs:line 1729
--- End of stack trace from previous location ---]]></stack-trace>
        </failure>
      </test>

This is most likely caused by the fact that the newlines in the content of the sources in AnalyzerTest.TestState.Sources, CodeFixTest.FixedState.Sources, and CodeFixTest.BatchFixedState.Sources are not normalized to Environment.NewLines.

Reproduction:

1. git clone https://github.com/dotnet/runtime from WSL or any setup where the default line ending is <LF>
2. Move the repo to a Windows system (where Environment.NewLine is <CRLF>)
3. Build the repo with ./build.cmd clr+libs+libs.tests -rc Release,
4. Run the tests with .\build.cmd -subset clr+libs+libs.tests -test -c Release
5. Observe that LibraryImportGenerator.Unit.Tests fail

[danmoseley]

The suggestion here is that this EqualOrDiff code be insensitive to LF vs CRLF in order to be robust to Git line ending settings when the "expected" output is in the repo.

[sharwell]

The current test framework behavior is by design. It is desirable for test authors to be alerted to cases where code fixes produce code with unexpected line endings. If a repository wishes to deviate from this behavior, it can adjust the construction mechanism to normalize the line endings at the time the test instance is created.

[sharwell]

@danmoseley Here's an example of normalization:

var code = @"
// Something ...
".ReplaceLineEndings();

Even if the file is checked out with LF line endings on a CRLF system, the call to ReplaceLineEndings() will ensure that code contains a string with the expected values.