The .NET SDK Integration (Discussion)

[Nirmal4G]

Can I do this today?

<Project Sdk="Microsoft.NET.Test.Sdk">

	<PropertyGroup>
		<TargetFrameworks>net8;net8-windows</TargetFrameworks>
	</PropertyGroup>

	<PropertyGroup>
		<!-- MSTest, VSTest, NUnit, xUnit, ... -->
		<UnitTestFramework>xUnit</UnitTestFramework>
		<!-- MSCodeCoverage, Coverlet, ... -->
		<CoverageTestFramework>Coverlet</CoverageTestFramework>
		<!-- Benchmark.NET, ... -->
		<PerformanceTestFramework>Benchmark.NET</PerformanceTestFramework>
	</PropertyGroup>

	<PropertyGroup Condition="'$(TargetFramework)' == 'net8-windows'">
		<UseWinUI>true</UseWinUI>
		<TestRunner>VSTest</TestRunner> <!-- Default: new MSTest Runner -->
		<OutputType>AppContainerExe</OutputType>
	</PropertyGroup>

	<ItemGroup Condition="'$(TargetFramework)' == 'net8-windows'">
		<PackageReference Include="Microsoft.WindowsAppSDK" />
	</ItemGroup>

</Project>

Assuming Default Includes in the Test SDK

<Project>

	<!-- Implicitly Included in the Test SDK -->
	<ItemGroup>
		<_ImplicitFrameworkReference Include="Microsoft.NET.Testing.App" />
	</ItemGroup>

	<!-- Implicit References from the above Shared Framework -->

	<!-- MSTest Includes -->
	<ItemGroup Condition="'$(UnitTestFramework)' == 'MSTest'">
		<_ImplicitReference Include="System.Testing" />
		<_ImplicitUsing Include="System.Testing" />
	</ItemGroup>

	<!-- VSTest (Legacy MSTest) Includes -->
	<ItemGroup Condition="'$(UnitTestFramework)' == 'VSTest'">
		<_ImplicitReference Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
		<_ImplicitUsing Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
	</ItemGroup>

	<!-- NUnit Includes -->
	<ItemGroup Condition="'$(UnitTestFramework)' == 'NUnit'">
		<_ImplicitReference Include="NUnit" />
		<_ImplicitUsing Include="NUnit.Framework" />
	</ItemGroup>

	<!-- xUnit Includes -->
	<ItemGroup Condition="'$(UnitTestFramework)' == 'xUnit'">
		<_ImplicitReference Include="xUnit" />
		<_ImplicitUsing Include="Xunit" />
	</ItemGroup>

	<!-- Similar Includes for Coverage and Performance frameworks -->

</Project>

Here's a crazy thought

The .NET currently doesn't have a default testing framework in BCL, so why not donate testfx to dotnet and make it into a base framework, say, System.Testing (with customizable AppHost, Async Testing, proper API and runtime hooks for benchmarks, coverage, reporting, tracing, exception handling, etc...).

The basic testing infrastructure, say, System.Testing (not just unit tests but also code coverage, performance (#2340), functional and end to end testing) so that other testing frameworks can extend it instead of competing with it. The basic premise is that the feature set of v1 of this should enable the testing of dotnet/runtime and siblings without extending too much.

I'm thinking along the lines of Microsoft.NET.Test.App or Microsoft.NET.Testing.App framework packages. It'll come with all the benefits of framework packages. Then an SDK package (Microsoft.NET.Test.Sdk) that comes inbox in .NET SDK. The main reason why I want this instead of bunch of library references is simplicity and maintainability when doing tests separately from build in isolated environment (no networking).

Some background on Testing Framework package requirement

We have certain (legacy) projects where tests are kept separately from the source. Only unit tests covering the source are included within the source repo. Every other test case is added on user report, vulnerability reports, etc... So, you can safely say that there are millions of test cases added over the years. We don't ever remove them even if they become redundant. Thus, building those tests and running them along with the source takes hours. That adds up with every commit. So, we test them asynchronously and separately from the main build only on hot path code change, before deployment, on private and public test environments, and before production deployment. This is what we call a build-less deployment testing. It has its ups and downs but in our case it works.

Here comes the problem: Since we are building the tests separately, we need to deploy both binaries, test data to several pre-configured environments which puts pressure on our internal network. So, in order to avoid network congestion, first we had a solution that restored the assemblies from deps.json but that was not feasible in the long run for several reasons. So, now, we build empty projects (another repo) with copy-local dependencies converted to Shared Framework package manually and pre-deployed in those environments once for every version update. This sort of mimics the built-in framework model and also enables us to do auditing. Recently we moved towards MSTest Runner as the default, which already removed a bunch of dependencies, custom code and simplified our environments.

Problem solved right? Nope not even close, as this also comes with a cost of maintaining and synchronising across all the repos. The actual binaries built from source are around 10-15. The rest of them are static versioned binaries that just need to be there in the environment. We tried using sharedfx Sdk that dotnet/runtime uses but it was too complicated for our use case. I just wanted a simple shared framework for our dependencies. So, As I said before, we package them manually.

The shared framework for our tests contains our own test abstractions and dependencies like MSTest, xUnit, Coverlet and Benchmark.NET, etc.... It'll be better if an all-encompassing Testing framework was included in NET SDK. That way we don't have to redistribute our base testing libraries every single time. If anyone says NuGet, yes, we do use a private feed but only during development and Source Build CI.

Note

I'm aware of some aspects of what is available today and what works now. So, this is more of a high-level direction than product feedback.

[Nirmal4G]

Looping @ViktorHofer in for thoughts on this with respect to the .NET Runtime.

[Evangelink]

Hi @Nirmal4G,

I won't lie, this is definitely my personal vision and goal, but it needs a lot of convincing in management and other teams :) When working on the naming for the new platform Microsoft.Testing.Platform/Microsoft.Testing.Extensions.XXX we did have some chat about the possibility to be part of System.Testing but this was denied back then.

About the example of <UnitTestFramework> property, I just want to make a clarification because there is often confusion (for a good reason, as things are hidden and not well-explained). In .NET testing, there are 2 components:

1. The platform/runner: this is the part of the code that is responsible for all the common infra, tooling and orchestration. This can be either a platform open to various test frameworks (e.g. VSTest, Microsoft.Testing.Platform) or something tighted to a given test framework (e.g. xunit.runner.console, NUnite Lite).

2. The test framework: the library responsible of defining the API and conventions on how to structure your tests (attributes, fixtures, assertions...).

3. Optionally, there is a test adapter that is doing the glue between the platform/runner and the test framework.

[Nirmal4G]

I do understand the differences between the runner/platform, adapter and framework terminologies. Thanks for writing it out.

For providing an example above, I didn't know what to name it. Initially, I thought Platform and Library. But they were either too broad or too narrow of a term and they didn't sound right with CodeCoverage and Benchmark.NET. Since it's also a user facing property, I decided to go with Framework.

we did have some chat about the possibility to be part of System.Testing but this was denied back then.

Any particular reason why it was denied? I don't think it could be because of API stability or design. If so, they could not have approved System.CommandLine! 😂