Avoid guid at the end of outputDirectory

[MarcoRossignoli]

On Coverlet we have some users complain about the dynamic folder name generated for reports file.
I did some check on code and seem that guid is "mandatory" and not removable.

https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Common/DataCollection/DataCollectionAttachmentManager.cs#L101-L125

Could be useful add a parameter, maybe in runsetting like <DeterministicOutputdirectory>true/false</DeterministicOutputdirectory> and in that case avoid to concat guid.ToString().

Dynamic folder makes CI more complex.

Issues refs coverlet-coverage/coverlet#500 coverlet-coverage/coverlet#767

cc: @DaleyKD

[nohwnd]

I don't know the reasons for the guid but I would rather avoid yet another option. I see the coverage file described in the TRX output, but even thought there is a href to it, it does not contain the guid. I also don't see the guid anywhere else in the repo. Imho the best course of action is to provide a way that will allow the test result to be programatically processed.

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

[nohwnd]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

[MarcoRossignoli]

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

Looking at publish result task https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml seem that a glob filter is used

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: '**/*.trx'
    failTaskOnFailedTests: true

[MarcoRossignoli]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

I'm ok with file format, but non deterministic path add some work to do in some scenario, tipical "find right file and move to".

[nohwnd]

I meant it more in the direction of returning a simple object that can be programatically processed to get to the info, instead of forcing the naming conventions to be deterministic. For example in Pester I simply return a result object that the caller can process. Here we could do something similar. Say you'd want to upload your CC file:

$result = dotnet test --collect="Code Coverage" ....params.... --json-result 
($result | ConvertFrom-Json).Attachements | where Type -eq "Code Coverage" | Invoke-RestMethod -Method PUSH

This would be the end product where we would have to come up with the result etc... So a more achievable goal would be to hookup the trx file and read the path to the CC file from there if it was correct.

[MarcoRossignoli]

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

If I don't use powershell what does --json-result produce?Is it a json on standard output?

[nohwnd]

Yes, it was just an exam

…

________________________________ Od: Marco Rossignoli <[email protected]> Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest <[email protected]> Kopie: Jakub Jareš <[email protected]>; Comment <[email protected]> Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378) $result = dotnet test --collect="Code Coverage" ....params.... --json-result Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2378 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA>.

[nohwnd]

Yes json, it was just an example to show a concept. I am not forcing powershell on anyone.

…

________________________________ Od: Marco Rossignoli <[email protected]> Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest <[email protected]> Kopie: Jakub Jareš <[email protected]>; Comment <[email protected]> Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378) $result = dotnet test --collect="Code Coverage" ....params.... --json-result Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2378 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA>.

[MarcoRossignoli]

What if we have more than one collector https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#enabledisable-a-datacollector?
A json array with different Type(collector friendly name)?

[ViktorHofer]

Just stumbled upon this.

I don't know the reasons for the guid but I would rather avoid yet another option.

I disagree. VSTest should respect the specified ResultsDirectory and CoverageFileName for data collectors and not create intermediate directories between. I can do that for loggers already so I would argue DataCollectors to behave the same:

<RunSettings>
  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>
  <LoggerRunSettings>
    <Loggers>
      <Logger uri="logger://Microsoft/TestPlatform/TrxLogger/v1">
        <Configuration>
          <LogFileName>testResults.trx</LogFileName>
        </Configuration>
      </Logger>
      <Logger uri="logger://Microsoft/TestPlatform/HtmlLogger/v1">
        <Configuration>
          <LogFileName>testResults.html</LogFileName>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>

I would expect the following behavior:

• LogFileName inside a DataCollector configuration node is respected:

<DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <CoverageFileName>coverage.xml</CoverageFileName>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

• The specified ResultsDirectory is respected without any intermediate folders in it.

  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>

In this example you can even see that coverage is explicitly listed in the comment (which I took from the docs). Specifying the log file name seems to have worked in the past with the pre-coverlet code coverage collector: #1957.

[ViktorHofer]

I just observed another misbehavior:

The coverage report is created twice:

• vihofer_DESKTOP-803H9R0_2020-04-22_06_59_42\In\DESKTOP-803H9R0\coverage.opencover.xml
• 4a408eed-656b-435d-a540-30c285d1c57a\coverage.opencover.xml

[MarcoRossignoli]

The coverage report is created twice:

@ViktorHofer I think it's a dup of #2334

[ViktorHofer]

You are right :)

[ViktorHofer]

FWIW I currently need to clean the TestResults directory manually before VSTest is invoked to make sure that ReportGenerator picks the right coverage results file later. Ideally we would not need to do this.

[paulomorgado]

The directory where the coverage results will be published is required to exist. Otherwise it throws:

Data collector 'Code Coverage' message: Data collector caught an exception of type 'Microsoft.VisualStudio.Coverage.VanguardException': 'Running event not received from CodeCoverage.exe. Check eventlogs for failure reason.'. More details: ..

That's also bad.

[ps2goat]

I think I see the purpose for the dynamic folder that is appended-- if you run it at the root level against a solution with multiple test projects, each project is in its own dynamic GUID folder.

A better solution would be to at least use the project name in this scenario. These names are predictable and should be unique:
TestResults/ProjectA/<coveragefile>.xml
TestResults/Projectb/<coveragefile>.xml

...unless there's some weird nesting in folders that would allow duplicate project names. That sounds like an edge case that the user would have to fix, though, and in that case you could use GUID folders per duplicated project or something.

[ArchieCoder]

Spent the evening doing tests here and there trying to understand why I was getting random folders. Glad that I finally found this, but sad this is not implemented. This is definitively not CI/CL friendly.

[hauntingEcho]

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder. Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

[tonyhallett]

dotnet test standard output provides the generated folder name

"Starting test execution, please wait...\r\nLogging Vstest Diagnostics in file: C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\diagnostics.log\r\nA total of 1 test files matched the specified pattern.\r\n\r\n
Attachments:\r\n C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\**7ba6447d-a89f-4836-bffc-aeb4799e48ab\coverage.cobertura.xml**r\nPassed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 7 ms - XUnitTestProject1.dll (netcoreapp3.1)"

The diagnostic log also has the generated folder name

TpTrace Verbose: 0 : 4712, 10, 2021/02/21, 14:58:38.317, 1073211811845, vstest.console.dll, DataCollectionRequestSender.SendAfterTestRunStartAndGetResult : Received message: (DataCollection.AfterTestRunEndResult) -> [
{
"Uri": "datacollector://Microsoft/CoverletCodeCoverage/1.0",
"DisplayName": "XPlat code coverage",
"Attachments": [
{
"Description": "",
"Uri": "file://C:/Users/tonyh/Source/Repos/DataCollectorXUnit/XUnitTestProject1/bin/Debug/netcoreapp3.1/fine-code-coverage/coverage-tool-output/7ba6447d-a89f-4836-bffc-aeb4799e48ab/coverage.cobertura.xml"
}
]
}
]

Agree that should avoid the guid though.

[psvaiter]

Is there any solution found for this issue?
I'm suffering from the same problem, while msbuild integration allows a specification of ouput dir. Why not vstest?

[ArchieCoder]

@psvaiter Here is my solution. I call this executable before using the coverage file. The tool search for the coverage file and copy at the specified file path.

`using System;
using System.IO;

namespace CopyCoverageFile
{
class Program
{
static void Main(string[] args)
{
if (args.Length == 2)
{
var files = Directory.GetFiles(args[0], "*.coverage", SearchOption.AllDirectories);

            var path = Path.Combine(args[1], "YOUR_TEST.coverage");

            Directory.CreateDirectory(args[1]);

            File.Copy(files[0], path);

            Console.WriteLine("Is coverage created? " + File.Exists(path));
        }
        else if (args.Length == 1)
        {
            Console.WriteLine($"Is new {args[0]} exist? " + File.Exists(args[0]));
        }
    }
}

}`

[Cobster]

Hoping this may help a few people out there, I use the following snippet of PowerShell in my development scripts to extract the unit test results paths and feed them to report generator.

$TestOutput = dotnet test --collect "XPlat Code Coverage"
$TestReports = $TestOutput | Select-String coverage.cobertura.xml | ForEach-Object { $_.Line.Trim() } | Join-String -Separator ';'
dotnet reportgenerator "-reports:$TestReports" "-targetdir:./CoverageReport" "-reporttype:Html"

[psvaiter]

@ArchieCoder I tried to use MSBuild integration and that worked as expected. Besides that I could use the Fine Code Coverage extension for Visual Studio that seems to depend on coverlet.msbuild package.

Here's the command I run:

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=TestResults\

This way I stopped using the vstest.

[keegandent]

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder. Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

I am right there with you. I'm not sure what genius thought an artifact directory's name should be non-deterministic, but it's completely ludicrous to me. Nobody has, from my reading, presented a single good argument as to why you'd want a GUID for this. I could see the argument for a timestamp, but honestly, if somebody wants their artifacts in a GUID-named folder, make them do it themselves as its a lot easier to go deterministic->random than vice-versa. I'm not sure why those who want consistency are the ones being punished.

[MarcoRossignoli]

Related issue #2334 (comment)

[ViktorHofer]

@MarcoRossignoli @nohwnd is someone actively working on this? It's still causing unreasonable pain and the issue is upvoted quite a lot. Also, the milestone seems to be incorrect as 17.1 already shipped.

[Evangelink]

@ViktorHofer We are not working on this. I will bring it up at tomorrow's standup.

[crojewsk]

Any progress? Everything that needed to be said was said in this topic. User should have an option to set static output directory and given the feedback, pretty sure this should be the default behavior.

Remember that you can always work incrementally on the final solution. First integrate the proposal found here (or anything similar) and then follow up with something neater if an additional option is not something you guys are aiming for.

If this issue is gone, please ignore. Using VS 2022 17.3.1 and the problem is still reproducing with dotnet test.

[crojewsk]

Provided (another?) example of run script workarounding this issue. Tested on my Win10 with msys2 and Debian9.

[ArchieCoder]

I believe the pseudo code for the solution that Microsoft could do is the following code. I have an hard time thinking it would be more complex than this.

var fixedDirectory = String.Empty;

if (args.Contains(“/fixed_directory”))
fixedDirectory = GetValue(“/fixed_directory”);

// Regular code

if (!string.IsNullOrEmpty(fixedDirectory))
File.Copy(guidDirectoryFileName, Path.Combine(fixedDirectory, Path.GetFileName(guidDirectoryFileName));

[bhehe]

Well, I've stumbled into this & the related issues as I'm encountering the same problems. I kept seeing duplicate files in the differing folder names (guid -vs- generated name). A bit saddened to see this has been an ongoing issue for what looks to be 2+ years now.

So far, I've seen the suggestion of parsing the TRX file to get the associated coverage file may be an avenue (but definitely not preferred) and then it seems like the other option is that if we know the folder-naming convention for the UserName + HostName prefix variant we could possibly just ignore/purge those as known dupes.

What I wanted to report & share (and maybe get an answer for) is that I have observed some very odd behavior along these lines; In my case I have 6 test projects and I use the --results-directory argument to point to my desired output path at the repo root (TestResults) along with the --logger "trx;LogFileName=MyProject.trx" argument and that does yield the correct/expected behavior.

But what I've noticed in regard to the code coverage result files is that the GUID-based directory is always present, and the file is always there, -but- I have seen the other User/Host name prefixed folders NOT get emitted reliably. And it's random it seems as I've not identified any pattern as to when it fails to emit that variant.

So, add me as +1 anxiously awaiting a good solution to this problem space. I'd very much prefer to have more control over both the location where the coverage file is output and the name of it without the need to detect & handle dupes. I realize this may require an effort by both the msft/vstest team and the coverlet team as they're just build atop the provided plumbing.

[bhehe]

After investigating if parsing the TRX file was something I wanted to pursue using, it looks like that is going to prove to be inadequate as well. In my case the path being mentioned in the 'attachments' is using the host-name - not the GUID-based folders. So, it's still ambiguous and as I was sharing in my case, I've seen those UserName+HostName folders not getting created reliably. Seems like I'm basically stuck with the 1 option of just ignoring/purging the folders that follow the UserName+HostName convention.

One question I do have: One of my key concerns is that when you process/analyze the code coverage files there is the notion of a 'hit count' and these dupes would be incrementing that count when they are merged - yielding misleading metrics potentially. Is that a correct assumption?

Here's an example of the contents from my TRX file.

<CollectorDataEntries> <Collector agentName="GWXXC09508" uri="datacollector://microsoft/CoverletCodeCoverage/1.0" collectorDisplayName="XPlat code coverage"> <UriAttachments> <UriAttachment> <A href="GWXXC09508\coverage.cobertura.xml"></A> </UriAttachment> </UriAttachments> </Collector> </CollectorDataEntries>

[stan-sz]

Sharing the below workaround. Note the single /*/ glob that will resolve to a direct child folder where the test attachment exists. The other xml files (from other test projects) exist in deeper folders.

 - task: PublishCodeCoverageResults@1
   displayName: Publish code coverage from $(Agent.TempDirectory)/*/*.cobertura.xml
   inputs:
     codeCoverageTool: Cobertura
     summaryFileLocation: $(Agent.TempDirectory)/*/*.cobertura.xml

[th3oth3rjak3]

Hoping this may help a few people out there, I use the following snippet of PowerShell in my development scripts to extract the unit test results paths and feed them to report generator.

$TestOutput = dotnet test --collect "XPlat Code Coverage"
$TestReports = $TestOutput | Select-String coverage.cobertura.xml | ForEach-Object { $_.Line.Trim() } | Join-String -Separator ';'
dotnet reportgenerator "-reports:$TestReports" "-targetdir:./CoverageReport" "-reporttype:Html"

I was absolutely struggling with this and I'm so glad that you pasted this in here. Thanks for an easy fix!

[jachstet-sea]

Instead of using a PowerShell snippet to do the operation "find the file in a random directory and copy it to someplace known", one can simply use a tool that can copy with "wildcard source paths". However, neither copy, xcopy nor robocopy seem to support that. Funnily, scp does. And it should be part of any modern Windows installation.

So, make sure you give a known output file name and use scp:

vstest.console.exe --logger:trx;LogFileName=%WORKSPACE%\UnitTestResults.trx;verbosity=detailed ^
--collect:"Code Coverage;Format=Xml;CoverageFileName=vstest.coveragexml" /EnableCodeCoverage ^
<YOUR UNIT TEST FILES HERE>

REM Copy the code coverage file to where it is expected
scp TestResults\*\vstest.xml vstest.coveragexml

Note, how the Code Coverage collector doesn't even respect the CoverageFileName since it simply changes the file extension... Way to go!

[jachstet-sea]

@Evangelink: What did mentioning this problem in your standup on 2022-08-04 bring as a result?

[Evangelink]

Hi @jachstet-sea,

We discussed these issues and we cannot do much because any change will have consequences on a lot of people. We will keep this issue open in case we can plan some breaking changes in some version.

[jachstet-sea]

We discussed these issues and we cannot do much because any change will have consequences on a lot of people. We will keep this issue open in case we can plan some breaking changes in some version.

Thanks for the update :)

[Evangelink]

Sorry we cannot offer anything better at the moment

[ArchieCoder]

@Evangelink If adding a parameter would cause the command line tool to break (which I doubt really), did your team consider fork this project and remove the guid creation?

[0xced]

Too bad this has not been addressed. In an attempt to workaround this issue and get deterministic output paths I wrote a an MSBuild target that moves files and deletes the directory with the guid. And of course I shot myself in the feet, deleted the wrong directory and now I'm running recovery software. I'm crossing fingers that I can recover my git repository where I had a lot of local branches (and stashes).

[ViktorHofer]

We discussed these issues and we cannot do much because any change will have consequences on a lot of people

@Evangelink did you consider adding a parameter that allows deterministic output paths, i.e. VSTestEnableDeterministicOutputPath that people can then set to true in msbuild or via a VSTest CLI argument?

[johnthagen]

After a long amount of searching around, what I had to do to solve this problem (alluded to in previous comments) is to add the following to my .Tests .csproj:

<PackageReference Include="coverlet.msbuild" Version="6.0.0"/>

And then I had to switch from the following command:

dotnet test --collect:"Code Coverage;Format=Cobertura;CoverageFileName=coverage.xml"

To:

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput="coverage.xml"

[Bitfiddler]

After a long amount of searching around, what I had to do to solve this problem (alluded to in previous comments) is to add the following to my .Tests .csproj:

<PackageReference Include="coverlet.msbuild" Version="6.0.0"/>

And then I had to switch from the following command:

dotnet test --collect:"Code Coverage;Format=Cobertura;CoverageFileName=coverage.xml"

To:

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput="coverage.xml"

As long as you don't mind this hanging over your head:
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md

If you use the coverlet.msbuild way of solving this problem, be aware that you may run into this and have to revert to using coverlet.collector and have to deal with the non-deterministic nonsense in the end anyway (like I did).

[jakubch1]

You can also use dotnet-coverage tool to resolve this issue:

dotnet-coverage collect -f cobertura -o "D:\coverage\report.cobertura.xml" "dotnet test"

[jakubch1]

Check how to use dotnet-coverage tool in this scenario:
https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario25/README.md

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform