Inconsistencies between nunit3-console and running the project via Visual Studio #1208
Comments
|
Some miscellaneous observations. Regarding running under VS versus under NUnit3Console... Of course it's a problem if the console runner can't run your tests, but it's not surprising if one works and the other doesn't... they are different pieces of software after all, written and maintained by different people. :-) Of course both the console runner and the NUnit adapter use the NUnit engine to run tests, but it's likely that they are using two different versions of the engine and calling it in different ways. When you use --inprocess, you are telling NUnit to try to load a .NET 5.0 test into a process running .NET Framework. That can't possibly work. Unfortunately, the actual error isn't reported very well. Anyway, don't try to run in the process unless you have a test assembly that targets the .NET Framework. I suggest simplifying things so we can try to debug this... Remove the reference to the NUnit3TestAdapter, which is not used by the console runner but may actually cause an engine version conflict. Use the test assembly (dll) as your argument rather than the csproj file so we are not using the project loader extension. Use the --trace:Debug argument and examine the log files to see if one of them tells you what is actually happen in the reflected code. Let me know what you learn and we'll take it from there. |
|
Hey, thanks for the explanations :) I did as you suggested and tried pointing at the There is a lot of mentions of frameworks and runtimes and SDKs and I'm a bit confused by all these things when it comes to |
|
The log you posted is the log for the main process running You need to examine all the logs to find any Error lines, which give a reason for the crash. |
|
Oh, my I see. Here is the log there is one error in the log: Not sure what that means though. And just to make sure there is no misunderstanding. The console |
|
The third and fifth lines from the bottom point you to yet another separate
log file. I don't see a file name there, however. Just make sure you start
out with no *.log files and it will be clear. There are generally three
logs: 1) for the console runner itself 2) for the agent 3) for the tests.
It's that third log which may show more exactly what has happened.
It is clear from the agent log that your test assembly was loaded
successfully. So all the other assembly references were satisfied. That
suggests that you are using some sort of dynamic loading in the test
itself, which is failing.
…On Mon, Aug 15, 2022 at 6:17 PM Oscar Aguinagalde ***@***.***> wrote:
Oh, my I see. Here is the log nunit-agent_128164.log:
InternalTrace: Initializing at level Debug
17:32:15.960 Info [ 1] NUnitTestAgent: Agent process 128164 starting
17:32:15.966 Debug [ 1] ServiceManager: Added ExtensionService
17:32:15.966 Debug [ 1] ServiceManager: Added DriverService
17:32:15.966 Info [ 1] NUnitTestAgent: Initializing Services
17:32:15.966 Info [ 1] ServiceManager: Initializing ExtensionService
17:32:15.968 Info [ 1] ExtensionService: Scanning nunit.engine.core assembly for extension points
17:32:15.970 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/NUnitV2Driver, Type=NUnit.Engine.Extensibility.IFrameworkDriver
17:32:15.972 Info [ 1] ExtensionService: Scanning nunit.engine.api assembly for extension points
17:32:15.972 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/TypeExtensions/IService, Type=NUnit.Engine.IService
17:32:15.972 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/TypeExtensions/ITestEventListener, Type=NUnit.Engine.ITestEventListener
17:32:15.972 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/TypeExtensions/IDriverFactory, Type=NUnit.Engine.Extensibility.IDriverFactory
17:32:15.972 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/TypeExtensions/IProjectLoader, Type=NUnit.Engine.Extensibility.IProjectLoader
17:32:15.972 Info [ 1] ExtensionService: Found Path=/NUnit/Engine/TypeExtensions/IResultWriter, Type=NUnit.Engine.Extensibility.IResultWriter
17:32:15.987 Info [ 1] ExtensionService: Processing file C:\Users\user\.nuget\packages\nunit.consolerunner\3.15.2\tools\agents\net5.0\nunit.agent.addins
17:32:15.990 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.consolerunner\3.15.2\tools' for extensions.
17:32:15.990 Info [ 1] ExtensionService: Processing file C:\Users\user\.nuget\packages\nunit.consolerunner\3.15.2\tools\nunit.console.nuget.addins
17:32:15.992 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitprojectloader\3.6.0\tools' for extensions.
17:32:16.026 Info [ 1] ExtensionService: .NET Core runners require .NET Core or .NET Standard extension for C:\Users\user\.nuget\packages\nunit.extension.nunitprojectloader\3.6.0\tools\nunit-project-loader.dll
17:32:16.026 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitv2driver\3.8.0\tools' for extensions.
17:32:16.026 Info [ 1] ExtensionService: Processing file C:\Users\user\.nuget\packages\nunit.extension.nunitv2driver\3.8.0\tools\nunit.v2.driver.addins
17:32:16.027 Info [ 1] ExtensionService: .NET Core runners require .NET Core or .NET Standard extension for C:\Users\user\.nuget\packages\nunit.extension.nunitv2driver\3.8.0\tools\nunit.v2.driver.dll
17:32:16.027 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitv2resultwriter\3.6.0\tools' for extensions.
17:32:16.027 Info [ 1] ExtensionService: .NET Core runners require .NET Core or .NET Standard extension for C:\Users\user\.nuget\packages\nunit.extension.nunitv2resultwriter\3.6.0\tools\nunit-v2-result-writer.dll
17:32:16.027 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.extension.teamcityeventlistener\1.0.7\tools' for extensions.
17:32:16.028 Info [ 1] ExtensionService: .NET Core runners require .NET Core or .NET Standard extension for C:\Users\user\.nuget\packages\nunit.extension.teamcityeventlistener\1.0.7\tools\teamcity-event-listener.dll
17:32:16.028 Info [ 1] ExtensionService: Scanning directory 'C:\Users\user\.nuget\packages\nunit.extension.vsprojectloader\3.8.0\tools' for extensions.
17:32:16.028 Info [ 1] ExtensionService: .NET Core runners require .NET Core or .NET Standard extension for C:\Users\user\.nuget\packages\nunit.extension.vsprojectloader\3.8.0\tools\vs-project-loader.dll
17:32:16.029 Warning [ 1] ExtensionService: Skipping directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitprojectloader\3.6.0\tools' because it was already visited.
17:32:16.029 Warning [ 1] ExtensionService: Skipping directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitv2driver\3.8.0\tools' because it was already visited.
17:32:16.029 Warning [ 1] ExtensionService: Skipping directory 'C:\Users\user\.nuget\packages\nunit.extension.nunitv2resultwriter\3.6.0\tools' because it was already visited.
17:32:16.029 Warning [ 1] ExtensionService: Skipping directory 'C:\Users\user\.nuget\packages\nunit.extension.teamcityeventlistener\1.0.7\tools' because it was already visited.
17:32:16.029 Warning [ 1] ExtensionService: Skipping directory 'C:\Users\user\.nuget\packages\nunit.extension.vsprojectloader\3.8.0\tools' because it was already visited.
17:32:16.029 Info [ 1] ServiceManager: Initializing DriverService
17:32:16.030 Debug [ 1] ServiceManager: Request for service ExtensionService satisfied by ExtensionService
17:32:16.031 Info [ 1] NUnitTestAgent: Starting RemoteTestAgent
17:32:16.033 Info [ 1] TestAgentTcpTransport: Connecting to TestAgency at 127.0.0.1:51134
17:32:16.039 Debug [ 1] NUnitTestAgent: Waiting for stopSignal
17:32:16.134 Error [ 4] ServiceManager: Requested service NUnit.Engine.ITestRunnerFactory was not found
17:32:16.137 Debug [ 4] ServiceManager: Request for service IDriverService satisfied by DriverService
17:32:16.139 Debug [ 4] DriverService: Trying NUnit3DriverFactory
17:32:16.139 Info [ 4] NUnit3DriverFactory: Using NUnitNetCore31Driver
17:32:16.140 Debug [ 4] NUnitNetCore31Driver: Loading C:\Users\user\git_projects\Server\ServiceTest\bin\Debug\net5.0\ServiceTest.dll
17:32:16.147 Debug [ 4] NUnitNetCore31Driver: Loaded C:\Users\user\git_projects\Server\ServiceTest\bin\Debug\net5.0\ServiceTest.dll
17:32:16.148 Debug [ 4] NUnitNetCore31Driver: Loaded nunit.framework
17:32:16.155 Debug [ 4] NUnitNetCore31Driver: Created FrameworkControler FrameworkController
17:32:16.155 Info [ 4] NUnitNetCore31Driver: Loading ServiceTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null - see separate log file
17:32:16.155 Debug [ 4] NUnitNetCore31Driver: Executing NUnit.Framework.Api.FrameworkController.LoadTests
17:32:16.189 Info [ 4] NUnitNetCore31Driver: Running ServiceTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null - see separate log file
17:32:16.613 Debug [ 1] NUnitTestAgent: Stop signal received
17:32:16.613 Info [ 1] NUnitTestAgent: Agent process 128164 exiting cleanly
there is one error in the log:
17:32:16.134 Error [ 4] ServiceManager: Requested service NUnit.Engine.ITestRunnerFactory was not found
Not sure what that means though. And just to make sure there is no
misunderstanding. The console nunit3-console is not crashing, its my test
code that is crashing. My code crashes because its looking for a package System.Security.Permissions,
Version=4.0.3.0 (the library MySQL.Data has System.Security.Permissions,
Version=4.0.7.0 as a dependency) and cant find it.
—
Reply to this email directly, view it on GitHub
<#1208 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAUN7BLO6CLOALDKVRNUIMTVZLT3HANCNFSM56KZJQJA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Managed to find the log file from the same execution when I created the issue! And yes, I remember step debugging my own code until it reaches a point in which it goes inside a function from the I've looked into it and I have found the following regarding assembly load (binding?) in C#...
I've checked and there is no Also I found out that there is certain |
|
I have noticed that I have also read that that path is the used to prove for assembly probing. I wonder if that could be related... And if so, is there a way to test that? |
|
I think I have found the issue, partially. I have found the way to check what are all the var allTheUsedDepsFiles = System.AppContext.GetData("APP_CONTEXT_DEPS_FILES");After checking that both running via VS and the nunit console this are the contents. And sure enough, I found the exact version of the assembly that visual studio was loading in the "System.Security.Permissions.dll": {
"assemblyVersion": "5.0.0.0",
"fileVersion": "5.0.20.51904"
},Here is the modules view when debugging via Visual Studio: It doesnt show the version, but it matches that of the What I dont know now, is whether nunit not using that Also... Why is it using the version 5 of the assembly when nuget has downloaded the correct version of the assembly specified as a dependency by the library? 🤔 |
|
We have a similar issue. #1202, #1203 and #1208 all seem to be caused by projects using additional .NET libraries, i.e. Windows Forms, WPF or ASP.NET. Since NUnit Console doesn't use those libraries they don't get added to its Our issue is with an unmanaged/native library. We are using a NuGet package (Esri.ArcGISRuntime) that has some unmanaged dependencies. As seems to be the convention, these get copied to a I can workaround the issue in a couple of ways:
I have found some explanation about how native assemblies are probed for (https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/default-probing), and potential ways of configuring it (https://github.com/dotnet/cli/blob/master/Documentation/specs/runtime-configuration-file.md), however I haven't been able to figure out any better workarounds than the two listed above. |
|
@cw397 Thanks! That's helpful information. |
|
A lot of different things have been discussed in this issue. Many have been solved and I would like to close it and create more narrow issues for what remains. There appear to be two remaining unclosed problems: 1. The problem with I believe that the fix to #1203 took care of this. @OAguinagalde can you verify using the MyGet release of the console runner? Version 3.16.0-dev00043 or a higher 3.16 version should have it. 2. The problem loading unmanaged libraries reported by @cw397 I haven't changed any code, which would effect this, so I'll make it a specific issue. Have I missed anything in the discussion? |
|
EDIT: Nevermind this, read the next comment! Hello, I cloned (3dd9e50 - I feel like I had this same issue when I first tested nunit-console and had a bunch of problems installing it, but dont remember how I fixed it back then... |
|
Edit: Ok, nevermid this as well, took another look, and once again I assumed things I shouldn't have assumed. This time I tried the version that you exactly said, the latest 3.16, and it works as you said, sorry!! And thanks so much for the support. I'll close this! Original commnet: |
|
@OAguinagalde I assume you closed this issue by mistake. You used version 4, which doesn't yet have the fix. You should use a version 3 drop, starting with 3.16.0-dev00043. The version 4 releases come from |
@CharliePoole I agree my issue is likely to have a separate underlying cause to the main one so am happy for you to make it its own issue. I think all the other issues are hopefully covered by your fix to #1203. |
|
@CharliePoole Sorry, my message was probably confusing with all the editing and stuff. The log using version 4 was the original message, which I then edited after retrying with the version 3.16. I confirmed it working and edited the message, without removing the log using version 4... In any case. It is 100% working now! I have been using since yesterday and its great! Feel free to close the issue :) |
|
I'm closing this as (mostly) completed. The remaining problem is described in #1253 |
|
This issue has been resolved in version 3.16.0 The release is available on: |
Running...
dotnet build nunit3-console.exe .\ServiceTest.csprojI get...
The same test run through
Visual Studio(orVisual Studio Code) runs without issues.I'm guessing this has to do with the version being used when run via
VSand when running it vianunit3-console. I'm not too familiar with all the versioning stuff with C# and .net.Checking the dependencies of
MySql.Dataproject inVSshows that it depends onSystem.Security.Permissions, Version=4.7.0but for some reason the error runningnunit3-consoleshows that it is looking forSystem.Security.Permissions, Version=4.0.3.0.I see that
nunit3-consoleis runningRuntime: .NET Framework CLR v4.0.3while my project is targettingnet5.0. Not sure if this is the problem, and if it is, I have tried using the--frameworkflag without success, whatever I write, it doesn't recognize it.This is the
csprojfile:some other notes on my system:
I installed
nunit3-consoleusingdotnet add package NUnit.Console --version 3.15.2. The project references:<PackageReference Include="NUnit" Version="3.13.2" />and<PackageReference Include="NUnit3TestAdapter" Version="4.0.0" />where added some time ago unlike the console which I got it today.Using
--inprocessgives this exception:My objective is to use
nunit3-console.The text was updated successfully, but these errors were encountered: