Spurious error is reported for non-test projects when running the solution #1465

WindingWinter · 2024-08-30T04:15:57Z

Referring to #1182. I have problem when I run tests using nunit console 3.18: spurious error is report for main project that targets .NET Standard 2.0

A description of my solution file: it consists of a .net standard 2.0 library, and a test library that consumes the main library. The test library is targeted to .net framework 4.8.

I download the nunit console 3.18. from here. And I run the nunit-console.exe from bin\net462 folder. When I run the nunit console, this is the output that I get:

Errors, Failures and Warnings

1) Failed : Nunit48.RunATest.NetStandard20AccountBalanceTest
  Expected: 45
  But was:  78.0d
   at Nunit48.RunATest.NetStandard20AccountBalanceTest() in E:\Net48NetStandard20\Nunit48\RunATest.cs:line 26

2) Failed : Nunit48.RunATest.RunB
  Expected: not equal to 42
  But was:  42
   at Nunit48.RunATest.RunB() in E:\Net48NetStandard20\Nunit48\RunATest.cs:line 34

3) Error : E:\Net48NetStandard20\NetStandard20Lib\bin\Debug\netstandard2.0\NetStandard20Lib.dll
System.ArgumentException : The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-2.0 net-3.0 net-3.5 net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-7.0 netcore-8.0
--ArgumentException
The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-2.0 net-3.0 net-3.5 net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-7.0 netcore-8.0
   at NUnit.Engine.Services.TestAgency.GetAgent(TestPackage package) in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Services\TestAgency.cs:line 71
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunnerIfNeeded() in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 250
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 138

Test Run Summary
  Overall result: Failed
  Test Count: 3, Passed: 1, Failed: 2, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 2, Errors: 0, Invalid: 0
  Start time: 2024-08-30 03:56:11Z
    End time: 2024-08-30 03:56:12Z
    Duration: 1.129 seconds

I can run the test library that consumes .net standard 2.0 code; as you can see above, the correct number of tests (3) are run and reported.

However, somehow the presence of NetStandard20Lib library is reported as an additional error

3) Error : E:\Net48NetStandard20\NetStandard20Lib\bin\Debug\netstandard2.0\NetStandard20Lib.dll
System.ArgumentException : The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-2.0 net-3.0 net-3.5 net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-7.0 netcore-8.0
--ArgumentException
The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-2.0 net-3.0 net-3.5 net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-7.0 netcore-8.0
   at NUnit.Engine.Services.TestAgency.GetAgent(TestPackage package) in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Services\TestAgency.cs:line 71
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunnerIfNeeded() in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 250
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in C:\Users\charlie\dev\NUnit\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 138

Which is completely spurious. The NetStandard20Lib.dll is the main library targetting .net standard 2.0. It is not supposed to be under testing. No idea why it is there at all.

I can confirm that running the nunit console in the nuget folder will also give me exactly the same error.

The text was updated successfully, but these errors were encountered:

WindingWinter · 2024-08-30T04:18:14Z

And this is the batch script I use to run the nunit console

call  "%NunitConsole%\nunit3-console.exe" "%slnFile%" /config:Debug --skipnontestassemblies --result="%testresultxml%"  --timeout=900000 > "%testresulttxt%"

nunitconsolerun.zip

WindingWinter · 2024-09-02T07:47:11Z

Here are my further findings.

It's not entirely clear what #1182 fixes and what it doesn't. On my side, the nunit console test runner won't crash-- that's great since it's now usable, but will report spurious errors with regards to .net standard 2.x DLLs as per above, even they are not test assemblies, and even I choose to skip them via the --skipnontestassemblies in the command line.

You might think that having one or two spurious errors are not big deal. But then, I have 10++ .net standard 2.0 DLLs in my solution, and for that solution, I run it a few times in my nightly builds, each time with a different filtering criteria. So it does clutter up my daily unit test reports. It's a huge eyesore for me to even sift through the reports for actual bugs and errors!

It seems that I will need to target it to .net framework or .net core, or both. So currently my workaround is to change in the main csproject file from

<TargetFrameworks>netstandard2.0</TargetFrameworks>

To

<TargetFrameworks>net48;net8.0</TargetFrameworks>

The alternative is to use nunitprojectloader and setup nunit project ( .nunit file), which I don't prefer due to the reasons that I will have to maintain one extra file.

Of course, given a choice I would still prefer to target to netstandard2.0-- it's clearer that way my design intention for the main projects. Alas this is the best I can do for now.

PS: is this a bug in vsprojectloader? Should someone move it to there?

CharliePoole · 2024-09-02T09:29:54Z

The message refers to netcore-4.5, which is NUnit's internal representation of the non-existent version 4.5 of .Net Core. It will come up if you use the TFM "net4.5" somewhere rather than "net45". It's hard to imagine how you could do that without having a compiler error. What the error message is really telling you is that no agent can be found to run the .Net Standard project.

The project loader was fixed in it's latest release, 3.9.0, to automatically skip non-test assemblies, so you don't actually need that option on the command line when running the solution. However, you should verify two things:

That you are actually running version 3.9.0 of the project loader.
That your .NET Standard project is really a non-test assembly.

For NUnit, a test assembly is any assembly, which references the NUnit Framework. It doesn't matter whether it has any tests. If you happen to reference the framework for some other reason, you can use the NonTestAssembly attribute to tell NUnit that the assembly is not intended to be a test assembly.

If none of those steps helps, we can keep this as a bug to fix. It will probably turn out to be a bug in the project loader, but it can stay here until I confirm that.

My advice in your case would be to stop running the solution file and just run your test assembly on the command-line. If there are multiple builds of the test assembly, just put them all on the command-line. This will serve as a workaround and maybe even as the true solution of your problem.

WindingWinter · 2024-09-02T09:53:42Z

@CharliePoole , I can confirm that

I don't set "net4.5" for TFM
I am actually running the project loader of version 3.9
That the .net standard project is really a non-test assembly with no reference to NUnit Framework

And I can really scale down a sample for test.

CharliePoole · 2024-09-04T16:25:19Z

@WindingWinter I changed the title to reflect that this issue is not really about .NET Standard 2.0 but about erroneously trying to examine non-test projects when using the VS Project loader extension to run from the solution file.

Most likely, this will be a bug in the extension but it could instead be in how the engine interacts with the extension. I'll leave it here until I am sure.

You mention above that you have a reduced solution, but I get errors trying to open the zip file. A repro I can use would be very helpful.

WindingWinter · 2024-09-05T08:30:35Z

@CharliePoole , I attached a new zip
nunitconsolerun_win.zip.

Previously I was zipping it using winrar, which is why you couldn't open it . Now you should be able to open it either way.

CharliePoole · 2024-09-05T10:44:37Z

@WindingWinter Thanks. I can open that one on pop-os and also on my windows vm.

I'll try to see first if this is actually a bug in the vs project loader, which should be skipping the non-test file automatically.

CharliePoole · 2024-09-05T23:49:02Z

I have added tests using VS Project loader 3.9.0 to the console runner's package tests. They confirm that the project loader is operating as designed. After the ProjectService calls the loader API ExpandProjectPackage method, both the original package and any subpackages created for projects under the solution all have the SkipNonTestAssemblies property set to true.

Note that the non-test projects are not simply dropped. They are still included as subpackages but the SkipNonTestAssemblies setting prevents the runner from attempting to execute them. Instead, they are given a special driver, the SkippedAssemblyFrameworkDriver, which merely reports them as skipped. We do it this way for two reasons:

So that the project loader does not have to understand how to locate tests and determine if we are dealing with a non-test assembly. This is left to the engine.
So that the skipped assembly will appear in the output rather than just disappearing.

Obviously an assembly requires a certain degree of analysis in order to determine that it's a non-test assembly. The exception we are seeing seems to indicate that certain checks are being made too early. That is, the engine should do what is needed to identify non-test projects first, before continuing with any deeper analysis. For example, we may not need to know the target runtime in order to ensure that we have a test project.

This is easy to state as a strategy but may be harder to implement. I'll take it one step at a time.

CharliePoole · 2024-09-05T23:52:48Z

@WindingWinter Could issue nunit/vs-project-loader#67 be affecting you? I worked around it in one of my own projects by duplicating elements in the project itself.

WindingWinter · 2024-09-06T01:59:25Z

@CharliePoole , it's not entirely clear to me how that nunit/vs-project-loader#67 affects anything ( I don't know nearly enough about the internal of nunit console or vs project loader to say).

The way I workaround the issue is that instead of using .net standard 2.0 as my target framework, I multi target the assembly to .net48 and .net80.

CharliePoole · 2024-09-06T03:36:50Z

@WindingWinter No need to understand internals. It's just a question of whether you use a Directory.Build.props file and what's in it.

WindingWinter · 2024-09-06T03:45:33Z

@CharliePoole , this I can answer. The answer is no I don't use Directory.Build.props at all at the moment.

CharliePoole · 2024-09-06T03:51:14Z

Good. That's one less thing to worry about. :-)

CharliePoole · 2024-09-06T03:58:50Z

I have replicated the problem with .NET Standard 2.0. The check and resulting exception are apparently made too early, before we have determined it's a non-test assembly.

CharliePoole · 2024-09-07T13:44:02Z

@WindingWinter This fix is available from myget either as a nuget package or a chococolatey package. Please try it and let me know how it works for you. You might also try that build for the problem in #1468. I'm doubtful, but there are substantial changes that could change the error even if it doesn't fix it.

CharliePoole · 2024-09-19T22:59:54Z

This issue has been resolved in version 3.18.2

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org

WindingWinter · 2024-09-20T01:48:24Z

@CharliePoole , I've downloaded and tried 3.18.2.

And this problem is still reproducible. Other issues like #1468 and #1469 seems to be solved.

CharliePoole · 2024-09-20T04:55:58Z

@WindingWinter I don't get it. I'm loading and running a solution with a netstandard project in my tests. Would you take a look at solution TestData.sln in the repository and see if you can determine how it is different from your own example?

WindingWinter · 2024-09-20T08:59:44Z

@CharliePoole ,

From your example here,

<TargetFrameworks>net462;netstandard2.0;netcoreapp3.1</TargetFrameworks>

Seems to work, but on my side, both plural and singular TargetFramework(s)

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>net48;netstandard2.0</TargetFrameworks>
  </PropertyGroup>

</Project>

and

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.1</TargetFramework>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>

don't work and will have the original error.

Do you think an attached zip example will help?

WindingWinter · 2024-09-20T09:04:51Z

Oh well, I can reproduce the problem with the zip file attached here.

Exactly the same error.

Here's the batch file if you are interested:

set NunitConsole=C:\Users\USER\.nuget\packages\nunit.consolerunner\3.18.2\tools
set DirRepo=E:\.net Sample Project\September 2024\nunitconsolerun
set slnFile=%DirRepo%\WPFNet48.sln
set testresultxml=%DirRepo%\TestResult.xml
set testresulttxt=%DirRepo%\TestResult.txt
set combinedTestResult=%DirRepo%\TestResultCombined.txt

rem call dotnet test --no-build "%slnFile%"  --settings "%~dp0test.runsettings"
call  "%NunitConsole%\nunit3-console.exe" "%slnFile%" --skipnontestassemblies --result="%testresultxml%"  > "%testresulttxt%"

CharliePoole · 2024-09-20T13:42:41Z

Is it possible the timeout issue is affecting you in this one as well?

If not, I'll download your entire solution and try that. For the release, I used it as a model but didn't use the whole thing.

WindingWinter · 2024-09-20T23:30:01Z

No, nothing to do with the timeout. As you can see from the batch script in my previous reply

…

On Fri, Sep 20, 2024, 9:43 PM CharliePoole ***@***.***> wrote: Reopened #1465 <#1465>. — Reply to this email directly, view it on GitHub <#1465 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADENIZUG3NQAGN5AMJMLJ4DZXQRAPAVCNFSM6AAAAABNLV5PN2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGM2DANJSGMYDAOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

CharliePoole · 2024-10-05T16:47:28Z

Example code is passing using the latest dev build (3.18.3-dev00009 +) from MyGet. No exception, two failures, one pass.

oscgu · 2024-11-14T17:50:14Z

I'm also getting the same error with 3.18.2 & 3.18.3, when trying to test a net48 WPF application that references a netstandard2.0 library, which has it's own test project in net8, even when the test count is 0 due to filters and the --skipnontestassemblies flag.

System.ArgumentException : The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-8.0
--ArgumentException
The netcore-4.5 framework is not available for X86=False.
Available frameworks: net-4.0 net-4.5 net-4.5.1 net-4.5.2 net-4.6 net-4.6.1 net-4.6.2 net-4.7 net-4.7.1 net-4.7.2 net-4.8 netcore-6.0 netcore-8.0
   at NUnit.Engine.Services.TestAgency.GetAgent(TestPackage package) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Services\TestAgency.cs:line 71
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunnerIfNeeded() in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 250
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 138

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2024-11-14 15:28:15Z
    End time: 2024-11-14 15:28:17Z
    Duration: 1.410 seconds

It does however work as expected on our old buildserver which uses the version 3.16.3, where it just outputs a warning as the overall result and no errors.

CharliePoole added the Bug label Sep 2, 2024

CharliePoole changed the title ~~nunit console 3.18: spurious error is report for main project that targets .NET Standard 2.0~~ Spurious error is reported for non-test projects when running the solution Sep 4, 2024

CharliePoole self-assigned this Sep 4, 2024

CharliePoole mentioned this issue Sep 4, 2024

Unable to run nunit-console with VSProjectLoader that has a WPF 8.0 project #1468

Closed

CharliePoole mentioned this issue Sep 7, 2024

Eliminate unnecessary errors when processing non-test assemblies #1473

Merged

CharliePoole closed this as completed in #1473 Sep 7, 2024

CharliePoole added this to the 3.18.2 milestone Sep 9, 2024

CharliePoole reopened this Sep 20, 2024

CharliePoole closed this as completed Oct 5, 2024

CharliePoole modified the milestones: 3.18.2, 3.18.3 Oct 5, 2024

oscgu mentioned this issue Nov 15, 2024

nunit-console 3.18.* fails on a mixed wpf net48 and netstandard2.0 project #1515

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spurious error is reported for non-test projects when running the solution #1465

Spurious error is reported for non-test projects when running the solution #1465

WindingWinter commented Aug 30, 2024 •

edited

Loading

WindingWinter commented Aug 30, 2024

WindingWinter commented Sep 2, 2024

CharliePoole commented Sep 2, 2024

WindingWinter commented Sep 2, 2024

CharliePoole commented Sep 4, 2024 •

edited

Loading

WindingWinter commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

WindingWinter commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

WindingWinter commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

CharliePoole commented Sep 7, 2024

CharliePoole commented Sep 19, 2024

WindingWinter commented Sep 20, 2024

CharliePoole commented Sep 20, 2024

WindingWinter commented Sep 20, 2024 •

edited

Loading

WindingWinter commented Sep 20, 2024

CharliePoole commented Sep 20, 2024

WindingWinter commented Sep 20, 2024 via email

CharliePoole commented Oct 5, 2024

oscgu commented Nov 14, 2024

Spurious error is reported for non-test projects when running the solution #1465

Spurious error is reported for non-test projects when running the solution #1465

Comments

WindingWinter commented Aug 30, 2024 • edited Loading

WindingWinter commented Aug 30, 2024

WindingWinter commented Sep 2, 2024

CharliePoole commented Sep 2, 2024

WindingWinter commented Sep 2, 2024

CharliePoole commented Sep 4, 2024 • edited Loading

WindingWinter commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

CharliePoole commented Sep 5, 2024

WindingWinter commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

WindingWinter commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

CharliePoole commented Sep 6, 2024

CharliePoole commented Sep 7, 2024

CharliePoole commented Sep 19, 2024

WindingWinter commented Sep 20, 2024

CharliePoole commented Sep 20, 2024

WindingWinter commented Sep 20, 2024 • edited Loading

WindingWinter commented Sep 20, 2024

CharliePoole commented Sep 20, 2024

WindingWinter commented Sep 20, 2024 via email

CharliePoole commented Oct 5, 2024

oscgu commented Nov 14, 2024

WindingWinter commented Aug 30, 2024 •

edited

Loading

CharliePoole commented Sep 4, 2024 •

edited

Loading

WindingWinter commented Sep 20, 2024 •

edited

Loading