No tests discovered #780

aggieben · 2017-05-03T14:18:57Z

Description

I converted a project to the new .csproj format recently as a part of an effort to port to .NET Core, and after I did that and got everything building, the vstest runner stopped being able to find tests in the project. The strangest thing to me is that I can run individual tests in VS2017, but when I try to run with "Run all tests", the runner fails to find any.

The Xunit runner does work, however. When I run dotnet xunit, it finds all the tests and I get the expected behavior.

Steps to reproduce

I'm not entirely sure what I did to get in this state, but the project is accessible here:
https://github.com/aggieben/dbup/tree/release/4.0.0

Access is open. Please git clone https://github.com/aggieben/dbup; cd dbup; git checkout release/4.0.0 to get the project(s) in question.

Expected behavior

VS2017 test runner should show discovered tests in test explorer, and when "run all tests" is selected, it should run the tests. Similarly, dotnet test should execute all the tests for both targets (netcoreapp1.0 and net46).

As mentioned above, the Xunit runner does produce the expected results.

Actual behavior

When running with dotnet test, this is the output:

C:\Users\ben\proj\oss\DbUp\src\dbup-tests [release/4.0.0 ≡]
λ dotnet test
Build started, please wait...
Engine\SqlScript.cs(15,33): warning CS0169: The field 'SqlScript.name' is never used [C:\Users\ben\proj\oss\DbUp\src\dbup-core\dbup-core.csproj]
Engine\SqlScript.cs(15,33): warning CS0169: The field 'SqlScript.name' is never used [C:\Users\ben\proj\oss\DbUp\src\dbup-core\dbup-core.csproj]
Helpers\NullJournalTests.cs(21,33): warning CS0618: 'SqlServerExtensions.SqlDatabase(SupportedDatabases, Func<IDbConnection>, string)' is obsolete: 'Pass connection string instead, then use .WithTransaction() and .WithTransactionPerScript() to manage connection behaviour' [C:\Users\ben\proj\oss\DbUp\src\dbup-tests\dbup-tests.csproj]
DeployChangesBuilderTests.cs(21,33): warning CS0618: 'SqlServerExtensions.SqlDatabase(SupportedDatabases, Func<IDbConnection>, string)' is obsolete: 'Pass connection string instead, then use .WithTransaction() and .WithTransactionPerScript() to manage connection behaviour' [C:\Users\ben\proj\oss\DbUp\src\dbup-tests\dbup-tests.csproj]
DeployChangesBuilderTests.cs(41,33): warning CS0618: 'SqlServerExtensions.SqlDatabase(SupportedDatabases, Func<IDbConnection>)' is obsolete: 'Pass connection string instead, then use .WithTransaction() and .WithTransactionPerScript() to manage connection behaviour' [C:\Users\ben\proj\oss\DbUp\src\dbup-tests\dbup-tests.csproj]
DeployChangesBuilderTests.cs(61,33): warning CS0618: 'SqlServerExtensions.SqlDatabase(SupportedDatabases, Func<IDbConnection>)' is obsolete: 'Pass connection string instead, then use .WithTransaction() and .WithTransactionPerScript() to manage connection behaviour' [C:\Users\ben\proj\oss\DbUp\src\dbup-tests\dbup-tests.csproj]
DeployChangesBuilderTests.cs(84,37): warning CS0618: 'SqlServerExtensions.SqlDatabase(SupportedDatabases, Func<IDbConnection>)' is obsolete: 'Pass connection string instead, then use .WithTransaction() and .WithTransactionPerScript() to manage connection behaviour' [C:\Users\ben\proj\oss\DbUp\src\dbup-tests\dbup-tests.csproj]
Build completed.

Test run for C:\Users\ben\proj\oss\DbUp\src\dbup-tests\bin\x64\Debug\netcoreapp1.0\dbup-tests.dll(.NETCoreApp,Version=v1.0)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test is available in C:\Users\ben\proj\oss\DbUp\src\dbup-tests\bin\x64\Debug\netcoreapp1.0\dbup-tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.



Test run for C:\Users\ben\proj\oss\DbUp\src\dbup-tests\bin\x64\Debug\net46\dbup-tests.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.9516916]   Discovering: dbup-tests
[xUnit.net 00:00:01.2040971]   Discovered:  dbup-tests
[xUnit.net 00:00:01.2348628] dbup-tests: Catastrophic failure: System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)
No test is available in C:\Users\ben\proj\oss\DbUp\src\dbup-tests\bin\x64\Debug\net46\dbup-tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

Environment

C:\Users\ben\proj\oss\DbUp\src\dbup-tests [release/4.0.0 ≡]
λ dotnet --info
.NET Command Line Tools (1.0.3)

Product Information:
 Version:            1.0.3
 Commit SHA-1 hash:  37224c9917

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.3

Test runner-related packages from the .csproj:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170106-08" />
    <PackageReference Include="xunit" Version="2.3.0-beta1-build3642" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.3.0-beta1-build1309" />
    <PackageReference Include="Shouldly" Version="2.8.1" />
    <PackageReference Include="TestStack.BDDfy" Version="4.3.1" />
    <!--<PackageReference Include="xunit.core" Version="2.2.0-beta2-build3300" />-->
    <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.0-beta1-build3642"/>
  </ItemGroup>

The text was updated successfully, but these errors were encountered:

bradwilson · 2017-05-03T16:31:23Z

This looks likely to be a bug in xUnit.net, not VSTest. There is a dictionary in our VS adapter which maps unique IDs to test cases that is probably being tripped up here. The question will be to figure out how you ended up with two test cases with the same unique ID,

aggieben · 2017-05-03T17:00:32Z

@bradwilson I see that from the output of the runner against the net46 target output, but do you think that's the case for the netcoreapp1.0 target as well?

bradwilson · 2017-05-03T17:32:22Z

Until I can poke around inside and see what's going on, I don't know. It could be that VSTest is handling the two scenarios differently (hiding the bug in one, exposing it in the other, but not running any tests in either case).

Faizan2304 · 2017-05-05T14:13:46Z

@aggieben Please use latest version of package Microsoft.NET.Test.Sdk which is 15.3.0-preview-20170502-03
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0-preview-20170502-03" />

With new package you will see that error message which adapter is sending is coming to console.

with latest version you will get following output:

PS C:\Enlistment\Customer\dbup\src\dbup-tests> dotnet test
Build started, please wait...
Build completed.

Test run for C:\Enlistment\Customer\dbup\src\dbup-tests\bin\Debug\netcoreapp1.0\dbup-tests.dll(.NETCoreApp,Version=v1.0)
Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170502-03
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:01.7501199]   Discovering: dbup-tests
[xUnit.net 00:00:02.1956541]   Discovered:  dbup-tests
[xUnit.net 00:00:02.3184782] dbup-tests: Catastrophic failure: System.ArgumentException: An item with the same key has already been added. Key: 9b4bc40fde591750f3097c5f4b22a610ad581cda
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)
No test is available in C:\Enlistment\Customer\dbup\src\dbup-tests\bin\Debug\netcoreapp1.0\dbup-tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Test run for C:\Enlistment\Customer\dbup\src\dbup-tests\bin\Debug\net46\dbup-tests.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170502-03
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.9311691]   Discovering: dbup-tests
[xUnit.net 00:00:01.2564212]   Discovered:  dbup-tests
[xUnit.net 00:00:01.3457598] dbup-tests: Catastrophic failure: System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)
No test is available in C:\Enlistment\Customer\dbup\src\dbup-tests\bin\Debug\net46\dbup-tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Faizan2304 · 2017-05-05T14:16:35Z

Only run all scenario ( From both commandline and visual studio) is busted here.
I am closing this bug as this is a bug in xunit adapter and @bradwilson is tracking it in xunit repo.

MarkHerhold · 2017-12-02T06:22:08Z

I am running into this exact issue except I am using the latest versions of everything. Is this still an outstanding issue with xunit or was it determined to be vstest?

smadala · 2017-12-02T06:37:11Z

@MarkHerhold Issue was determined as xunit adapter's. Can you share repro project, logs(https://github.com/Microsoft/vstest-docs/blob/master/docs/diagnose.md) and explicitly provide the versions of (dotnet --info, VS, xunit and Microsoft.NET.Test.Sdk)?

MarkHerhold · 2017-12-02T17:27:33Z

@smadala The strange thing is that the tests work just fine when run from VS Community Edition. Luckily, the project is open source and brand new, so that should make things simple.

Locally, I'm running VS Community Edition 2017 v14.4.5 (new install) with .NET Framework v4.7.02556 (came with new VS install).
xunit v2.3.1 defined here
Microsoft.NET.Test.Sdk v15.5.0 defined here

`dotnet --info` (local)

PS D:\badgeup-dotnet-core-client> dotnet --info
.NET Command Line Tools (2.0.3)

Product Information:
 Version:            2.0.3
 Commit SHA-1 hash:  12f0c7efcc

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.0.3\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.3
  Build    : a9190d4a75f4a982ae4b4fa8d1a24526566c69df

I also have the project running in AppVeyor with the same failures. Here's a link to the repo.

Logs of the tests working in VS Studio Community Edition:

[12/1/2017 9:52:22 PM Informational] ------ Load Playlist started ------
[12/1/2017 9:52:22 PM Informational] ========== Load Playlist finished (0:00:00.015711) ==========
[12/1/2017 9:52:50 PM Warning] The build was canceled or Failed.
[12/1/2017 9:52:52 PM Informational] ------ Discover test started ------
[12/1/2017 9:52:52 PM Error] The operation failed as details for project BadgeUpClient.Tests could not be loaded.
[12/1/2017 9:52:56 PM Informational] [xUnit.net 00:00:00.8590506]   Discovering: BadgeUpClient.Tests
[12/1/2017 9:52:56 PM Informational] [xUnit.net 00:00:01.4440466]   Discovered:  BadgeUpClient.Tests
[12/1/2017 9:52:58 PM Informational] ========== Discover test finished: 9 found (0:00:06.0468183) ==========
[12/1/2017 9:53:22 PM Informational] ------ Discover test started ------
[12/1/2017 9:53:24 PM Informational] [xUnit.net 00:00:00.4483696]   Discovering: BadgeUpClient.Tests
[12/1/2017 9:53:24 PM Informational] [xUnit.net 00:00:00.6953388]   Discovered:  BadgeUpClient.Tests
[12/1/2017 9:53:25 PM Informational] ========== Discover test finished: 9 found (0:00:02.6410116) ==========
[12/1/2017 9:53:25 PM Informational] ------ Run test started ------
[12/1/2017 9:53:26 PM Informational] [xUnit.net 00:00:00.3977474]   Discovering: BadgeUpClient.Tests
[12/1/2017 9:53:27 PM Informational] [xUnit.net 00:00:00.6313037]   Discovered:  BadgeUpClient.Tests
[12/1/2017 9:53:27 PM Informational] [xUnit.net 00:00:00.6358743]   Starting:    BadgeUpClient.Tests
[12/1/2017 9:53:27 PM Informational] [xUnit.net 00:00:01.5273870]   Finished:    BadgeUpClient.Tests
[12/1/2017 9:53:28 PM Informational] ========== Run test finished: 9 run (0:00:02.5004514) ==========

And here's the dotnet test --diag:log.txt file.

smadala · 2017-12-04T05:26:52Z

@MarkHerhold I couldn't repro it. See the below image. Share all log.*.txt files.

MarkHerhold · 2017-12-04T15:45:52Z

@smadala I see one thing you did differently is you cd'd to the Test project directory. Could you try running the tests at the directory above that?

Edit: Thanks for your help trying this! I'll collect the other log later today.

smadala · 2017-12-04T16:07:50Z

Running tests for sln, not yet supported from vstest(#1129). I tried running from above directory. See the below image. Did you face issue with dotnet test \path\to\BadgeUpClient.Tests.csproj, If so share log.txt and log.host.*.txt too.

MarkHerhold · 2017-12-04T23:10:33Z

@smadala Ah, that's the issue! Thanks so much for your help! 😀

bradwilson mentioned this issue May 3, 2017

No tests discovered xunit/xunit#1238

Closed

Faizan2304 self-assigned this May 4, 2017

Faizan2304 closed this as completed May 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No tests discovered #780

No tests discovered #780

aggieben commented May 3, 2017 •

edited

Loading

bradwilson commented May 3, 2017

aggieben commented May 3, 2017 •

edited

Loading

bradwilson commented May 3, 2017

Faizan2304 commented May 5, 2017

Faizan2304 commented May 5, 2017

MarkHerhold commented Dec 2, 2017

smadala commented Dec 2, 2017

MarkHerhold commented Dec 2, 2017 •

edited

Loading

smadala commented Dec 4, 2017

MarkHerhold commented Dec 4, 2017 •

edited

Loading

smadala commented Dec 4, 2017

MarkHerhold commented Dec 4, 2017

No tests discovered #780

No tests discovered #780

Comments

aggieben commented May 3, 2017 • edited Loading

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

bradwilson commented May 3, 2017

aggieben commented May 3, 2017 • edited Loading

bradwilson commented May 3, 2017

Faizan2304 commented May 5, 2017

Faizan2304 commented May 5, 2017

MarkHerhold commented Dec 2, 2017

smadala commented Dec 2, 2017

MarkHerhold commented Dec 2, 2017 • edited Loading

dotnet --info (local)

smadala commented Dec 4, 2017

MarkHerhold commented Dec 4, 2017 • edited Loading

smadala commented Dec 4, 2017

MarkHerhold commented Dec 4, 2017

aggieben commented May 3, 2017 •

edited

Loading

aggieben commented May 3, 2017 •

edited

Loading

MarkHerhold commented Dec 2, 2017 •

edited

Loading

`dotnet --info` (local)

MarkHerhold commented Dec 4, 2017 •

edited

Loading