Tests in runtime can't target older version of .NET

[ericstj]

Description

We have some nuget packages where it would be nice to test against older versions of .NET. I tried doing this but hit a few problems.

Reproduction Steps

Add $(NetCoreAppMinimum) to a test project and build and run tests.

Expected behavior

Tests should build and run against the older runtime - located from dotnet on the path.

Actual behavior

Tests fail to start

'"\dotnet.exe"' is not recognized as an internal or external command,  operable program or batch file.

This due to test script using %RUNTIME_PATH%\ but never setting it:

runtime/eng/testing/tests.targets

Lines 33 to 34 in 4777beb

	<RunScriptHostDir Condition="'$(RunScriptWindowsCmd)' == 'true'">%RUNTIME_PATH%\</RunScriptHostDir>
	<RunScriptHostDir Condition="'$(RunScriptWindowsCmd)' != 'true'">$RUNTIME_PATH/</RunScriptHostDir>

I can workaround this by setting

set RUNTIME_PATH=C:\Program Files\dotnet

Then I see

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

This is due to microsoft.dotnet.xunitconsolerunner only supporting latest.

So I worked around that with changes to

runtime/eng/testing/xunit/xunit.console.targets

Lines 45 to 51 in a3bf53f

	<PackageReference Include="Microsoft.DotNet.XUnitConsoleRunner"
	Version="$(MicrosoftDotNetXUnitConsoleRunnerVersion)"
	Condition="'$(TargetFrameworkIdentifier)' == '.NETCoreApp'" />
	<PackageReference Include="xunit.runner.console"
	Version="$(XUnitVersion)"
	GeneratePathProperty="true"
	Condition="'$(TargetFrameworkIdentifier)' == '.NETFramework'" />

I also needed to set XunitConsoleNetCore21AppPath since this wasn't set by the official package.

After that I got much closer - stuff was running. I noticed that some assemblies weren't copied though. If the assembly was a framework assembly it would not be copied (even though the locally built project references were newer than the 8.0 framework bits).

That's due to the hardcoding here;

runtime/eng/references.targets

Line 33 in a3bf53f

<Target Name="UpdateProjectReferencesWithPrivateAttribute"

Ideally we shouldn't be doing this, but instead flow the references through conflict resolution for it to do version comparisons. That's what will happen when users install the packages we produce. Conflict resolution doesn't work by default for project references dotnet/sdk#2674, so we might need to workaround that.

To unblock I just manually set Private="True" on the reference I cared about. After this I almost got things working, but RemoteExecutor seemed to be not using the application's deps file.

Here's some of the hacking: ericstj@7f0a1a8

Regression?

Nope, it's just runtime being 🌟 🌟 special🌟 🌟

Known Workarounds

See above

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

[ViktorHofer]

Duplicate of #54639