No coverage for async methods with RunMsCodeCoverage

[daryl0313]

Installed product versions

• Visual Studio: 2022 Professional
• This extension: 1.1.173

Description

Getting no coverage results for async methods with RunMsCodeCoverage set to Yes.

Steps to recreate

I created a simple solution that reproduces the issue for Framework and dotnet core project versions.

1. Clone this repo: https://github.com/daryl0313/FCCAsyncTestIssue
2. Verify that RunMsCodeCoverage is set to Yes
3. Run all tests

Current behavior

Only seeing coverage for synchronous methods in the coverage report and in the gutter in the editor. Seeing 3 covered lines for each test project in the report.

Expected behavior

See coverage for all methods in the coverage report and in the gutter in the editor. Should see 6 covered lines for each test project in the report.

Side Notes

• Sounds like it could be related to this: Async methods excluded from coverage after updating to 2.8.1 coverlet-coverage/coverlet#794 (comment)
  • I can't see the command that FCC is using to run the tests and generate the coverage xml file in the logs.
  • I don't see any option to not exclude that attribute. Even with ExcludeByAttribute cleared out, I get the same result.

Coverage Log

FCC Output

Fine Code Coverage : Ms code coverage

Fine Code Coverage : ReportGenerator Run Arguments [reporttype:Cobertura] 
"-targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\a0534286-1257-4f3c-ad4f-732653dc99c9\derek.batenhorst_DEV183_2022-06-15.11_36_17.cobertura.xml"
"-reporttypes:Cobertura"

Fine Code Coverage : ReportGenerator Run [reporttype:Cobertura]
2022-06-15T11:36:23: Arguments
2022-06-15T11:36:23:  -targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output
2022-06-15T11:36:23:  -reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\a0534286-1257-4f3c-ad4f-732653dc99c9\derek.batenhorst_DEV183_2022-06-15.11_36_17.cobertura.xml
2022-06-15T11:36:23:  -reporttypes:Cobertura
2022-06-15T11:36:24: Writing report file 'C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml'
2022-06-15T11:36:24: Report generation took 0.2 seconds

Fine Code Coverage : ReportGenerator Run Arguments [reporttype:HtmlInline_AzurePipelines] 
"-targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml"
"-plugins:C:\USERS\DEREK.BATENHORST\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_AF1CC7F5\EXTENSIONS\ZNCDRFMP.P1L\ReportGeneratorPlugins.dll"
"-reporttypes:FccLight"
"riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30"
"riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15"
"riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200"

Fine Code Coverage : ReportGenerator Run [reporttype:HtmlInline_AzurePipelines]
2022-06-15T11:36:24: Arguments
2022-06-15T11:36:24:  -targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output
2022-06-15T11:36:24:  -reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml
2022-06-15T11:36:24:  -plugins:C:\USERS\DEREK.BATENHORST\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_AF1CC7F5\EXTENSIONS\ZNCDRFMP.P1L\ReportGeneratorPlugins.dll
2022-06-15T11:36:24:  -reporttypes:FccLight
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200
2022-06-15T11:36:25: Writing report file 'C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\index.html'
2022-06-15T11:36:25: Report generation took 0.2 seconds

Fine Code Coverage : Processing cobertura

Fine Code Coverage : Processing report

Fine Code Coverage : ================================== DONE ==================================

[tonyhallett]

Sounds like it could be related to this: coverlet-coverage/coverlet#794 (comment)

No. As the coverage is not being collected by coverlet.

I believe that the problem is due to the version of ReportGenerator that FCC is currently using.

I am working on a feature branch https://github.com/tonyhallett/FineCodeCoverage/tree/webview2-reportgenerator-api that uses ReportGenerator 5.1.4 as a nuget reference compared to current that uses reportgenerator.exe 4.7.1. The feature branch does not have the issue.

If you look in ...FCCAsyncTestIssue-master\FCCAsyncTestIssue-master\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output

there is a Cobertura.xml file that ReportGenerator has generated from the ms cobertura.xml. You can see that the ms one is correct and the ReportGenerator one is not.

This will be rectified once I complete and push the feature.

If you need the fix now I can send you a vsix of the feature build but note the following:

It relies upon WebView2. That you have the Evergreen Runtime installed https://docs.microsoft.com/en-us/microsoft-edge/webview2/concepts/distribution#the-evergreen-runtime-distribution-mode
( When I finished the feature FCC will install it for you if necessary )

The report is not currently reflecting visual studio theming / environment fonts.

The report does not currently have sticky tables.

[JayOwl]

I'm not sure it's related to this issue but I got my async methods to be covered by changing the setting Tools > Options > Fine Code Coverage > AdjacentBuildOutput to true. Thanks Jeremy Thompson.

[viceice]

@JayOwl Are you using coverlet or Microsoft.CodeCoverage ? I think that workaround doesn't work for Microsoft.CodeCoverage

@tonyhallett Any chance to get a release with new feature? Or at least a prerelease to test 🙃

[tonyhallett]

@viceice

I think that workaround doesn't work for Microsoft.CodeCoverage
Agreed

The new feature now has the necessary code for the WebView2 runtime so what remains is the styling of the react report. I should be able to complete this in the next two weeks.

[viceice]

Sorry for noise. 🤗 Any news? It's pretty anoying to not see async function coverage. 😕

[tonyhallett]

Has been fixed 1f040cb but..........

@FortuneN
Your publisher access token has expired https://github.com/FortuneN/FineCodeCoverage/runs/8041220586?check_suite_focus=true

The new release is not my new feature of WebView2 + React report. This I will finish when I return from holiday.

To have ms coverage async methods working we cannot just update to the latest report generator version that fixed danielpalme/ReportGenerator#467 as an earlier commit removed ie and hence WebBrowser control support - danielpalme/ReportGenerator@f801790

So I have forked the repo at 4.7.1 and added the single line of code that fixes async. This code is https://github.com/FortuneN/FineCodeCoverage/blob/master/Shared%20Files/ZippedTools/reportGenerator.4.7.2.zip and https://github.com/FortuneN/FineCodeCoverage/blob/master/Shared%20Files/ZippedTools/ReportGenerator.Core.dll

So please update your access token !

[tonyhallett]

Although the vsix marketplace release github action did not occur, the release action did.

https://github.com/FortuneN/FineCodeCoverage/releases/tag/v1.1.181

So you can use
https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.181/FineCodeCoverage2022.vsix
https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.181/FineCodeCoverage.vsix

[viceice]

will test it on Monday ❤️

[FortuneN]

Done.
https://github.com/FortuneN/FineCodeCoverage/runs/8052519049?check_suite_focus=true

[viceice]

Publish was skipped, so still missing on store

[tonyhallett]

@FortuneN

https://github.com/FortuneN/FineCodeCoverage/runs/8188031339?check_suite_focus=true

add to marketplace

VSSDK: error VsixPub0031 : An error occurred while attempting to authenticate: VS30063: You are not authorized to access https://marketplace.visualstudio.com./

Once rectified the workflow cannot be re-run but needs to be triggered. Let me know when ready. Thanks

[tonyhallett]

https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage.vsix
https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage2022.vsix

This release fixes generic classes with ms code coverage.

[kakins]

https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage.vsix https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage2022.vsix

This release fixes generic classes with ms code coverage.

If I'm not mistaken then, this release also includes fixing async methods, correct? It appears to be working for me now

[tonyhallett]

@kakins correct

[viceice]

@FortuneN It's still not publishing to the store 😕

[FortuneN]

I've updated the token.
The only issue on my run is the version which i'm sure is a nothing-burger; right @tonyhallett ?
https://github.com/FortuneN/FineCodeCoverage/actions/runs/3023299512/jobs/4904846997

[tonyhallett]

@FortuneN
You cannot use workflow_dispatch. It needs to be triggered by pull request merge. I do one shortly.

At a later date can look at the actions further.

[tonyhallett]

@FortuneN
Still getting the access token issue https://github.com/FortuneN/FineCodeCoverage/actions/runs/3045484608/jobs/4907101471

Error: Error: Command failed: VsixPublisher.exe publish -payload D:\a\FineCodeCoverage\FineCodeCoverage\FineCodeCoverage\bin\Release\FineCodeCoverage.vsix -publishManifest D:\a\FineCodeCoverage\FineCodeCoverage\vs-market-place-manifest.json -personalAccessToken ***
VSSDK: error VsixPub0031 : An error occurred while attempting to authenticate: VS30063: You are not authorized to access https://marketplace.visualstudio.com./

[FortuneN]

Publish seems to be resolved.

Not sure about that last 'comment' bit

https://github.com/FortuneN/FineCodeCoverage/actions/runs/3077073438/jobs/4971676090

[tonyhallett]

Thanks. I will kick off the process again. At a later date the workflows can be revisited to support manual triggers.

…

On Tue, 13 Sept 2022 at 12:04, Fortune Ngwenya ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/5636175/189885287-e5e38417-6106-4831-b8e7-14d3a60ac213.png> I updated the token — Reply to this email directly, view it on GitHub <#270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWFCRUZEHNEEFNRZZ5JE3LV6BNUNANCNFSM5Y36PSMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>