Breaks when using reproducible builds

[AshleighAdams]

When attempting to mutation test something built with deterministic/continuous integration set in versions equal to or greater than 1.3.0, the test usually (but not always) fails with a varying error:

Unhandled exception.

no valid language detected || no valid csproj or fsproj was given

Expected behavior
Mutation test to complete.

Desktop (please complete the following information):

• OS: Windows & Linux, likely others
• Type of project: core
• Framework Version: net 5.0
• Stryker Version: >= 1.3.0

Additional context
Can be reproduced by cloning Verlite, setting the GITHUB_ACTIONS env var to true and launching the mutation test:

Ashleigh@Ashleigh-PC MINGW64 /c/Git/Verlite/tests/UnitTests (lock-stryker-version)
$ GITHUB_ACTIONS="true" dotnet stryker --reporter html --reporter progress --log-to-file

   _____ _              _               _   _ ______ _______  
  / ____| |            | |             | \ | |  ____|__   __| 
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |    
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |    
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |    
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|    
                   __/ |                                      
                  |___/                                       


Version: 1.3.1

[19:21:08 INF] Identifying project to mutate.
[19:21:10 INF] The project C:\Git\Verlite\src\Verlite.Core\Verlite.Core.csproj will be mutated.
[19:21:12 WRN] Buildalyzer could not find sourcefiles. This should not happen. Will fallback to filesystem scan. Please report an issue at github.
[19:21:13 INF] Analysis complete.
[19:21:13 INF] Building test project C:\Git\Verlite\tests\UnitTests\UnitTests.csproj (1/1)
[19:21:24 INF] Time Elapsed 00:00:16.4759913
Unhandled exception.

no valid language detected || no valid csproj or fsproj was given

log-20220210.txt
VsTest-log.host.22-02-10_19-12-32_80171_9.txt
VsTest-log.txt

[rouke-broersma]

This is actually way more concerning to me, as this fallback method is basically deprecated and we are intending to remove it:

[19:21:12 WRN] Buildalyzer could not find sourcefiles. This should not happen. Will fallback to filesystem scan. Please report an issue at github.

We will want to fix this for users using reproducible builds, fixing this will also fix the other errors. It seems like Buildalyzer cannot deal with projects using reproducible builds in CI (so in CI mode which as you said before makes things deterministic including normalizing source paths). The fix might be to simply delete the CI msbuild property from the buildalyzer analysis. We should discuss with upstream if this is something that should be done in general or if this is something that should be done on a case-by-case basis.

[0xced]

I'm currently trying to reproduce this issue on a local checkout of the repository and what's baffling is the non reproducible nature of this bug. Performing the following commands on a local checkout sometimes works and sometimes fails with the Buildalyzer could not find sourcefiles warning.

git clean -fdx
cd tests/UnitTests
GITHUB_ACTIONS="true" dotnet stryker --reporter html --reporter progress --log-to-file

And that's really sometimes, I can't reproduce this problem 100% of the time!

[0xced]

OK, so I found something very interesting: when selecting the IAnalyzerResult in ProjectFileReader.cs, the First() method is used.

if (targetFramework == null)
{
    return analyzerResults.First();
}

But the analyzerResults collection actually contains 2 elements, that's probably what explains the non reproducible nature of this bug.

One of them is the expected (valid) analyzer result and the other one is a broken analyzer result containing no references nor source files. I noticed that for the broken result, the TargetFramework property is null. That's probably an issue to report to the Buildalyzer repository.

As a workaround, using analyzerResults.First(e => e.TargetFramework != null) instead of analyzerResults.First() seems to solve this issue.

[0xced]

Further investigating into Buildalyzer, I figured it will probably difficult to fix this issue in Buildalyzer because a null target framework is expected for C++ projects.

// use an empty string if no target framework was found, for example in case of C++ projects with VS >= 2022

So filtering out analyzer results that don't have a target framework as hinted in my previous comment is probably the way to go.

[0xced]

I just opened #1900 to address this issue.

[0xced]

I opened phmonte/Buildalyzer#198 on the Buildalyzer repository.