New article that details unit test code coverage using coverlet (#18955)

* Initial bits of new article * Fix linting issue * Correct link * Updates * Final updates * Fix md error * Apply suggestions from code review Co-authored-by: Tom Dykstra <[email protected]> * Many more updates * Massive updates, more like a tutorial now * Added a lot and managed updates * Minor tweaks * Updates for acrolinx * Another minor update * Apply suggestions from code review Co-authored-by: Tom Dykstra <[email protected]> * Updates from peer review. * Proper casing * Final change Co-authored-by: Tom Dykstra <[email protected]>
dotnet · Jun 17, 2020 · cfdd645 · cfdd645
1 parent 876af04
commit cfdd645
Show file tree

Hide file tree

Showing 5 changed files with 310 additions and 1 deletion.
diff --git a/docs/core/testing/media/test-report.png b/docs/core/testing/media/test-report.png
diff --git a/docs/core/testing/order-unit-tests.md b/docs/core/testing/order-unit-tests.md
@@ -79,4 +79,4 @@ To order tests explicitly, NUnit provides an [`OrderAttribute`](https://github.c
 ## Next Steps
 
 > [!div class="nextstepaction"]
-> [Unit testing best practices](unit-testing-best-practices.md)
+> [Unit test code coverage](unit-testing-code-coverage.md)
diff --git a/docs/core/testing/unit-testing-best-practices.md b/docs/core/testing/unit-testing-best-practices.md
@@ -44,6 +44,12 @@ Writing tests for your code will naturally decouple your code, because it would
 - **Self-Checking**. The test should be able to automatically detect if it passed or failed without any human interaction.
 - **Timely**. A unit test should not take a disproportionately long time to write compared to the code being tested. If you find testing the code taking a large amount of time compared to writing the code, consider a design that is more testable.
 
+## Code coverage
+
+A high code coverage percentage is often associated with a higher quality of code. However, the measurement itself *cannot* determine the quality of code. Setting an overly ambitious code coverage percentage goal can be counterproductive. Imagine a complex project with thousands of conditional branches, and imagine that you set a goal of 95% code coverage. Currently the project maintains 90% code coverage. The amount of time it takes to account for all of the edge cases in the remaining 5% could be a massive undertaking, and the value proposition quickly diminishes.
+
+A high code coverage percentage is not an indicator of success, nor does it imply high code quality. It jusst represents the amount of code that is covered by unit tests. For more information, see [unit testing code coverage](unit-testing-code-coverage.md).
+
 ## Let's speak the same language
 The term *mock* is unfortunately very misused when talking about testing. The following defines the most common types of *fakes* when writing unit tests:
 

diff --git a/docs/core/testing/unit-testing-code-coverage.md b/docs/core/testing/unit-testing-code-coverage.md
@@ -0,0 +1,301 @@
+---
+title: Use code coverage for unit testing
+description: Learn how to use the code coverage capabilities for .NET unit tests.
+author: IEvangelist
+ms.author: dapine
+ms.date: 06/16/2020
+---
+
+# Use code coverage for unit testing
+
+Unit tests help to ensure functionality, and provide a means of verification for refactoring efforts. Code coverage is a measurement of the amount of code that is run by unit tests - either lines, branches, or methods. As an example, if you have a simple application with only two conditional branches of code (_branch a_, and _branch b_), a unit test that verifies conditional _branch a_ will report branch code coverage of 50%.
+
+This article discusses the usage of code coverage for unit testing with Coverlet and report generation using ReportGenerator. While this article focuses on C# and xUnit as the test framework, both MSTest and NUnit would also work. Coverlet is an [open source project on GitHub](https://github.com/coverlet-coverage/coverlet) that provides a cross platform code coverage framework for C#. [Coverlet](https://dotnetfoundation.org/projects/coverlet) is part of the .NET foundation. Coverlet collects Cobertura coverage test run data, which is used for report generation.
+
+Additionally, this article details how to use the code coverage information collected from a Coverlet test run to generate a report. The report generation is possible using another [open source project on GitHub - ReportGenerator](https://github.com/danielpalme/ReportGenerator). ReportGenerator converts coverage reports generated by Cobertura among many others, into human readable reports in various formats.
+
+## System under test
+
+The "system under test" refers to the code that you're writing unit tests against, this could be an object, service, or anything else that exposes testable functionality. For the purpose of this article, you'll create a class library that will be the system under test, and two corresponding unit test projects.
+
+### Create a class library
+
+From a command prompt in a new directory named `UnitTestingCodeCoverage`, create a new .NET standard class library using the [`dotnet new classlib`](../tools/dotnet-new.md#classlib) command:
+
+```dotnetcli
+dotnet new classlib -n Numbers
+```
+
+The snippet below defines a simple `PrimeService` class that provides functionality to check if a number is prime. Copy the snippet below and replace the contents of the *Class1.cs* file that was automatically created in the *Numbers* directory. Rename the *Class1.cs* file to *PrimeService.cs*.
+
+```csharp
+namespace System.Numbers
+{
+    public class PrimeService
+    {
+        public bool IsPrime(int candidate)
+        {
+            if (candidate < 2)
+            {
+                return false;
+            }
+
+            for (int divisor = 2; divisor <= Math.Sqrt(candidate); ++divisor)
+            {
+                if (candidate % divisor == 0)
+                {
+                    return false;
+                }
+            }
+            return true;
+        }
+    }
+}
+```
+
+> [!TIP]
+> It is worth mentioning the that `Numbers` class library was intentionally added to the `System` namespace. This allows for <xref:System.Math?displayProperty=fullName> to be accessible without a `using System;` namespace declaration. For more information, see [namespace (C# Reference)](../../csharp/language-reference/keywords/namespace.md).
+
+### Create test projects
+
+Create two new **xUnit Test Project (.NET Core)** templates from the same command prompt using the [`dotnet new xunit`](../tools/dotnet-new.md#test) command:
+
+```dotnetcli
+dotnet new xunit -n XUnit.Coverlet.Collector
+```
+
+```dotnetcli
+dotnet new xunit -n XUnit.Coverlet.MSBuild
+```
+
+Both of the newly created xUnit test projects need to add a project reference of the *Numbers* class library. This is so that the test projects have access to the *PrimeService* for testing. From the command prompt, use the [`dotnet add`](../tools/dotnet-add-reference.md) command:
+
+```dotnetcli
+dotnet add XUnit.Coverlet.Collector\XUnit.Coverlet.Collector.csproj reference Numbers\Numbers.csproj
+```
+
+```dotnetcli
+dotnet add XUnit.Coverlet.MSBuild\XUnit.Coverlet.MSBuild.csproj reference Numbers\Numbers.csproj
+```
+
+The *MSBuild* project is named appropriately, as it will depend on the [coverlet.msbuild](https://www.nuget.org/packages/coverlet.msbuild) NuGet package. Add this package dependency by running the [`dotnet add package`](../tools/dotnet-add-package.md) command:
+
+```dotnetcli
+cd XUnit.Coverlet.MSBuild && dotnet add package coverlet.msbuild && cd ..
+```
+
+The previous command changed directories effectively scoping to the *MSBuild* test project, then added the NuGet package. When that was done, it then changed directories, stepping up one level.
+
+Open both of the *UnitTest1.cs* files, and replace their contents with the following snippet. Rename the *UnitTest1.cs* files to *PrimeServiceTests.cs*.
+
+```csharp
+using System.Numbers;
+using Xunit;
+
+namespace XUnit.Coverlet
+{
+    public class PrimeServiceTests
+    {
+        readonly PrimeService _primeService;
+
+        public PrimeServiceTests() => _primeService = new PrimeService();
+
+        [
+            Theory,
+            InlineData(-1), InlineData(0), InlineData(1)
+        ]
+        public void IsPrime_ValuesLessThan2_ReturnFalse(int value) =>
+            Assert.False(_primeService.IsPrime(value), $"{value} should not be prime");
+
+        [
+            Theory,
+            InlineData(2), InlineData(3), InlineData(5), InlineData(7)
+        ]
+        public void IsPrime_PrimesLessThan10_ReturnTrue(int value) =>
+            Assert.True(_primeService.IsPrime(value), $"{value} should be prime");
+
+        [
+            Theory,
+            InlineData(4), InlineData(6), InlineData(8), InlineData(9)
+        ]
+        public void IsPrime_NonPrimesLessThan10_ReturnFalse(int value) =>
+            Assert.False(_primeService.IsPrime(value), $"{value} should not be prime");
+    }
+}
+```
+
+### Create a solution
+
+From the command prompt, create a new solution to encapsulate the class library and the two test projects. Using the [`dotnet sln`](../tools/dotnet-sln.md) command:
+
+```dotnetcli
+dotnet new sln -n XUnit.Coverage
+```
+
+This will create a new solution file name `XUnit.Coverage` in the *UnitTestingCodeCoverage* directory. Add the projects to the root of the solution.
+
+## [Linux](#tab/linux)
+
+```dotnetcli
+dotnet sln XUnit.Coverage.sln add **/*.csproj --in-root
+```
+
+## [Windows](#tab/windows)
+
+```dotnetcli
+dotnet sln XUnit.Coverage.sln add (ls **/*.csproj) --in-root
+```
+
+---
+
+Build the solution using the [`dotnet build`](../tools/dotnet-build.md) command:
+
+```dotnetcli
+dotnet build
+```
+
+If the build is successful, you've created the three projects, appropriately referenced projects and packages, and updated the source code correctly. Well done!
+
+## Tooling
+
+There are two types of code coverage tools:
+
+- **DataCollectors:** DataCollectors monitor test execution and collect information about test runs. They report the collected information in various output formats, such as XML and JSON. For more information, see [your first DataCollector](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md).
+- **Report generators:** Use data collected from test runs to generate reports, often as styled HTML.
+
+In this section, the focus is on data collector tools. To use Coverlet for code coverage, an existing unit test project must have the appropriate package dependencies, or alternatively rely on [.NET global tooling](../tools/global-tools.md) and the corresponding [coverlet.console](https://www.nuget.org/packages/coverlet.console) NuGet package.
+
+## Integrate with .NET test
+
+The xUnit test project template already integrates with [coverlet.collector](https://www.nuget.org/packages/coverlet.collector) by default.
+From the command prompt, change directories to the *XUnit.Coverlet.Collector* project, and run the [`dotnet test`](../tools/dotnet-test.md) command:
+
+```dotnetcli
+cd XUnit.Coverlet.Collector && dotnet test --collect:"XPlat Code Coverage"
+```
+
+> [!NOTE]
+> The `"XPlat Code Coverage"` argument is a friendly name that corresponds to the data collectors from Coverlet. This name is required but is case insensitive.
+
+As part of the `dotnet test` run, a resulting *coverage.cobertura.xml* file is output to the *TestResults* directory. The XML file contains the results. This is a cross platform option that relies on the .NET Core CLI, and it is great for build systems where MSBuild is not available.
+
+Below is the example *coverage.cobertura.xml* file.
+
+```xml
+<?xml version="1.0" encoding="utf-8"?>
+<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1592248008"
+          lines-covered="12" lines-valid="12" branches-covered="6" branches-valid="6">
+  <sources>
+    <source>C:\</source>
+  </sources>
+  <packages>
+    <package name="Numbers" line-rate="1" branch-rate="1" complexity="6">
+      <classes>
+        <class name="Numbers.PrimeService" line-rate="1" branch-rate="1" complexity="6"
+               filename="Numbers\PrimeService.cs">
+          <methods>
+            <method name="IsPrime" signature="(System.Int32)" line-rate="1"
+                    branch-rate="1" complexity="6">
+              <lines>
+                <line number="8" hits="11" branch="False" />
+                <line number="9" hits="11" branch="True" condition-coverage="100% (2/2)">
+                  <conditions>
+                    <condition number="7" type="jump" coverage="100%" />
+                  </conditions>
+                </line>
+                <line number="10" hits="3" branch="False" />
+                <line number="11" hits="3" branch="False" />
+                <line number="14" hits="22" branch="True" condition-coverage="100% (2/2)">
+                  <conditions>
+                    <condition number="57" type="jump" coverage="100%" />
+                  </conditions>
+                </line>
+                <line number="15" hits="7" branch="False" />
+                <line number="16" hits="7" branch="True" condition-coverage="100% (2/2)">
+                  <conditions>
+                    <condition number="27" type="jump" coverage="100%" />
+                  </conditions>
+                </line>
+                <line number="17" hits="4" branch="False" />
+                <line number="18" hits="4" branch="False" />
+                <line number="20" hits="3" branch="False" />
+                <line number="21" hits="4" branch="False" />
+                <line number="23" hits="11" branch="False" />
+              </lines>
+            </method>
+          </methods>
+          <lines>
+            <line number="8" hits="11" branch="False" />
+            <line number="9" hits="11" branch="True" condition-coverage="100% (2/2)">
+              <conditions>
+                <condition number="7" type="jump" coverage="100%" />
+              </conditions>
+            </line>
+            <line number="10" hits="3" branch="False" />
+            <line number="11" hits="3" branch="False" />
+            <line number="14" hits="22" branch="True" condition-coverage="100% (2/2)">
+              <conditions>
+                <condition number="57" type="jump" coverage="100%" />
+              </conditions>
+            </line>
+            <line number="15" hits="7" branch="False" />
+            <line number="16" hits="7" branch="True" condition-coverage="100% (2/2)">
+              <conditions>
+                <condition number="27" type="jump" coverage="100%" />
+              </conditions>
+            </line>
+            <line number="17" hits="4" branch="False" />
+            <line number="18" hits="4" branch="False" />
+            <line number="20" hits="3" branch="False" />
+            <line number="21" hits="4" branch="False" />
+            <line number="23" hits="11" branch="False" />
+          </lines>
+        </class>
+      </classes>
+    </package>
+  </packages>
+</coverage>
+```
+
+> [!TIP]
+> As an alternative, you could use the MSBuild package if your build system already makes use of MSBuild. From the command prompt, change directories to the *XUnit.Coverlet.MSBuild* project, and run the `dotnet test` command:
+>
+> ```dotnetcli
+> dotnet test --collect:"XPlat Code Coverage"
+> ```
+>
+> The resulting *coverage.cobertura.xml* file is output.
+
+## Generate reports
+
+Now that you're able to collect data from unit test runs, you can generate reports using [ReportGenerator](https://github.com/danielpalme/ReportGenerator). To install the [ReportGenerator](https://www.nuget.org/packages/dotnet-reportgenerator-globaltool) NuGet package as a [.NET global tool](../tools/global-tools.md), use the [`dotnet tool install`](../tools/dotnet-tool-install.md) command:
+
+```dotnetcli
+dotnet tool install -g dotnet-reportgenerator-globaltool
+```
+
+Run the tool and provide the desired options, given the output *coverage.cobertura.xml* file from the previous test run.
+
+```console
+reportgenerator
+"-reports:Path\To\TestProject\TestResults\{guid}\coverage.cobertura.xml"
+"-targetdir:coveragereport"
+-reporttypes:Html
+```
+
+After running this command, an HTML file represents the generated report.
+
+:::image type="content" source="media/test-report.png" lightbox="media/test-report.png" alt-text="Unit test-generated report":::
+
+## See also
+
+- [Visual Studio unit test cover coverage](/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested)
+- [GitHub - Coverlet repository](https://github.com/coverlet-coverage/coverlet)
+- [GitHub - ReportGenerator repository](https://github.com/danielpalme/ReportGenerator)
+- [ReportGenerator project site](https://danielpalme.github.io/ReportGenerator)
+- [.NET Core CLI test command](../tools/dotnet-test.md)
+
+## Next Steps
+
+> [!div class="nextstepaction"]
+> [Unit testing best practices](unit-testing-best-practices.md)
diff --git a/docs/core/toc.yml b/docs/core/toc.yml
@@ -431,6 +431,8 @@ items:
     href: testing/selective-unit-tests.md
   - name: Order unit tests
     href: testing/order-unit-tests.md
+  - name: Unit test code coverage
+    href: testing/unit-testing-code-coverage.md
   - name: Unit test published output
     href: testing/unit-testing-published-output.md
   - name: Live unit test .NET Core projects with Visual Studio