dotnet test for multi target net461 and netcoreapp3.1 success in windows but fail in Ubuntu 20.04 with error: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel

[moh-hassan]

Environment

VS code
Version: 1.46.1
OS: Windows_NT x64 10.0.19041
Working in WSL 2 environment (Ubuntu 20.04, Mono is installed)

Steps to reproduce

A multi-target sdk-style Xunit test project

<TargetFrameworks>net461;netcoreapp3.1</TargetFrameworks>

Package reference:

<ItemGroup>
    <PackageReference Include="FluentAssertions" Version="5.4.1" />    
    <PackageReference Include="xunit" Version="2.4.0" />    
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" /> 
   <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.6.1" />    
  </ItemGroup>

I followed the instructions Getting Started with xUnit.net Multi-targeting with non-Windows OSes

The test success locally in windows 10 x64 using vs code:

dotnet test

When running the test remotly in WSL 2 environment it fail for net461 with error:

Could not load type of field 'Xunit.Runner.VisualStudio.VsExecutionSink:recorder' (4) due to: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

If I add the next reference, test success

<PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="16.6.1" />  

Should I add reference to Microsoft.TestPlatform.ObjectModel when runing in Linux environment in net461?
What the role of this package?

[nohwnd]

I tried to reproduce and did not see the issue. Is there anything more to how you build / run the tests?

[moh-hassan]

Can you try to run test for this project
Only I use 'dotnet test' from terminal in vscode when remotely in wsl 2.

[moh-hassan]

Also, I run dotnet test in native Ubuntu 18.04.2 LTS and the same error in net461 is fired.
here more details (the same in wsl2 and native Ubuntu client)

Click to expand Server stack trace!

Server stack trace:
at (wrapper managed-to-native) System.RuntimeTypeHandle.type_is_assignable_from(System.Type,System.Type)
at System.RuntimeTypeHandle.CanCastTo (System.RuntimeType type, System.RuntimeType target) [0x00000] in :0
at System.RuntimeType.IsAssignableFrom (System.Type c) [0x00020] in :0
at System.RuntimeTypeHandle.IsContextful (System.RuntimeType type) [0x00000] in :0
at System.RuntimeType.IsContextfulImpl () [0x00000] in :0
at System.Type.get_IsContextful () [0x00000] in :0
at System.Runtime.Remoting.RemotingServices.Unmarshal (System.Runtime.Remoting.ObjRef objectRef, System.Boolean fRefine) [0x00041] in :0
at System.Runtime.Remoting.RemotingServices.Unmarshal (System.Runtime.Remoting.ObjRef objectRef) [0x00000] in :0
at System.Runtime.Remoting.Messaging.CADMessageBase.UnmarshalArgument (System.Object arg, System.Collections.ArrayList args) [0x0003d] in :0
at System.Runtime.Remoting.Messaging.CADMessageBase.UnmarshalArguments (System.Object[] arguments, System.Collections.ArrayList args) [0x00011] in :0
at System.Runtime.Remoting.Messaging.CADMethodCallMessage.GetArgs (System.Collections.ArrayList args) [0x00000] in :0
at System.Runtime.Remoting.Messaging.MethodCall..ctor (System.Runtime.Remoting.Messaging.CADMethodCallMessage msg) [0x0001e] in :0
at System.AppDomain.ProcessMessageInDomain (System.Byte[] arrRequest, System.Runtime.Remoting.Messaging.CADMethodCallMessage cadMsg, System.Byte[]& arrResponse, System.Runtime.Remoting.Messaging.CADMethodReturnMessage& cadMrm) [0x00012] in :0
at (wrapper remoting-invoke-with-check) System.AppDomain.ProcessMessageInDomain(byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&)
at System.Runtime.Remoting.Channels.CrossAppDomainSink.ProcessMessageInDomain (System.Byte[] arrRequest, System.Runtime.Remoting.Messaging.CADMethodCallMessage cadMsg) [0x0000d] in :0

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke (System.Runtime.Remoting.Proxies.RealProxy rp, System.Runtime.Remoting.Messaging.IMessage msg, System.Exception& exc, System.Object[]& out_args) [0x0014d] in :0
at (wrapper managed-to-native) System.Object.__icall_wrapper_mono_remoting_wrapper(intptr,intptr)
at (wrapper remoting-invoke) Xunit.Sdk.TestFrameworkExecutor1[Xunit.Sdk.IXunitTestCase].RunTests(System.Collections.Generic.IEnumerable1<Xunit.Abstractions.ITestCase>,Xunit.Abstractions.IMessageSink,Xunit.Abstractions.ITestFrameworkExecutionOptions)
at (wrapper xdomain-invoke) Xunit.Sdk.TestFrameworkExecutor1[Xunit.Sdk.IXunitTestCase].RunTests(System.Collections.Generic.IEnumerable1<Xunit.Abstractions.ITestCase>,Xunit.Abstractions.IMessageSink,Xunit.Abstractions.ITestFrameworkExecutionOptions)
at Xunit.Xunit2.RunTests (System.Collections.Generic.IEnumerable1[T] testCases, Xunit.Abstractions.IMessageSink messageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions executionOptions) [0x0000e] in <4d712162c0a34cb99c967a7a106084df>:0 at Xunit.XunitFrontController.RunTests (System.Collections.Generic.IEnumerable1[T] testMethods, Xunit.Abstractions.IMessageSink messageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions executionOptions) [0x00006] in <4d712162c0a34cb99c967a7a106084df>:0
at TestFrameworkExtensions.RunTests (Xunit.Abstractions.ITestFrameworkExecutor executor, System.Collections.Generic.IEnumerable`1[T] testCases, Xunit.IMessageSinkWithTypes executionMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions executionOptions) [0x00008] in <4d712162c0a34cb99c967a7a106084df>:0
at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly (Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, LoggerHelper logger, Xunit.Runner.VisualStudio.TestPlatformContext testPlatformContext, Xunit.Runner.VisualStudio.RunSettings runSettings, Xunit.IMessageSinkWithTypes reporterMessageHandler, Xunit.Runner.VisualStudio.AssemblyRunInfo runInfo) [0x00511] in <42d73e9a900048ada4679362440112c9>:0
No test is available in /home/user1/commandline28/commandline/tests/CommandLine.Tests/bin/Debug/net461/CommandLine.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Test Run Failed.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:.

Note: I checked this issue, but it is not relevant.
The solution in this issue is like my workaround solution.

Attach of linux session

[nohwnd]

I can see the same thing now. I would go with the workaround that you found. I couldn't think of any other way to solve this right now.

[moh-hassan]

What is the role of Microsoft.TestPlatform.ObjectModel in Linux?

[nohwnd]

That assembly holds all the common types. On windows it grabs it from the SDK where we ship testhost.exe, but on Linux that does not work for some reason.

[moh-hassan]

I installed xunit.console.exe v2.4.1 to run the test in Linux with Mono installed (Mono version 6.8.0.105).

Xunit test success
the next script is working:

mono path/to/xunit.console.exe   path/to/net461/test.dll       # net461 working

[moh-hassan]

@nohwnd
Also, If the package Microsoft.VisualStudio.TestPlatform.ObjectModel is added as a reference, test success.
The package Microsoft.VisualStudio.TestPlatform.ObjectModel is used by Xunit project and it's one of the dependencies of Microsoft.TestPlatform.ObjectModel.
It provides interfaces such as ITestExecutor and ITestDiscoverer for implementing custom unit test adapters.

It was requested by @bradwilson, the author of Xunit, here

The referenced assembly is a contract assembly which is expected to be provided by the version of Visual Studio (and/or VSTest) which is running the tests.

This package is used also by netcoreapp3.1 but it's loaded during netcoreapp3.1 build but it's not loaded during net461 build.
This part of .deps.json for netcoreapp3.1 project:

"Microsoft.TestPlatform.ObjectModel/16.6.1": {
        "dependencies": {
          "NuGet.Frameworks": "5.0.0"
        },
        "runtime": {
          "lib/netstandard2.0/Microsoft.TestPlatform.CoreUtilities.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          },
          "lib/netstandard2.0/Microsoft.TestPlatform.PlatformAbstractions.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          },
          "lib/netstandard2.0/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          }
        },

I noticed that the package has no project reference and it's not maintained since 2017, but the package is included in netcoreapp3.1 as version 16.6.1.

Suggested solution
The package Microsoft.TestPlatform.ObjectModel should be part of the dependencies of the package Microsoft.NET.Test.Sdk in net4x and it can solve many like issues in Linux/Mac with Mono installed.

[bradwilson]

@nohwnd I know you closed this as "Won't Fix", but this continues to be a sporadic problem on non-Windows machines trying to run net4xx unit tests. My current Ubuntu 22.04 WSL image suffers from this.

Repro.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
    <PackageReference Include="xunit" Version="2.5.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.5.1" />
  </ItemGroup>

</Project>

UnitTest1.cs

using Xunit;

public class UnitTest1
{
    [Fact]
    public void Test1()
    {
    }
}

Result:

Adding a references to Microsoft.TestPlatform.ObjectModel fixes the problem:

I've seen evidence through web searches that this happens to people using xUnit.net, NUnit, and MSTest. This leads me to believe that Microsoft.NET.Test.Sdk should include Microsoft.TestPlatform.ObjectModel as a dependency for .NET Framework as well as .NET Core. Can you explain why you've marked this as "won't fix"? How does VSTest provide this dependency when it's not part of the compilation steps? Any idea why is it sometimes broken, sometimes not?

If the decision remains to leave this as "won't fix", then I'm considering adding Microsoft.TestPlatform.ObjectModel as a NuGet dependency for xunit.runner.visualstudio (based on the version I used at compile time). This isn't something I want to do, and it's unclear to me whether doing this without also adding an identical version dependency for Microsoft.NET.Test.Sdk (something we don't reference, but is necessary to get tests to run) is asking for trouble. If I do just ObjectModel, then we can end up with version mismatches; if I do both, then we could potentially conflict with the user adding Sdk (which has been guidance for many years). On the other hand, if I don't do this, then I'm adding frustration to some users, who may abandon xUnit.net when it doesn't work without asking for help via an official channel.

Can you provide guidance on what the best way is for me to fix this for xUnit.net users in the safest possible way?

[bradwilson]

After mulling it over last night, and doing some testing w/rt to version mismatches (no issues I could uncover), I've decided to add the dependency to Microsoft.TestPlatform.ObjectModel. Whether this issue gets fixed here or not (which it absolutely should, IMO, since it has affected every unit test framework and not just xUnit.net), we cannot assume that users are diligently updating their reference to Microsoft.NET.Test.Sdk to grab the latest version. This reference is a "chore" reference that's left to the user to add in order to make unit testing work at all, and not something that developers interact with directly.

https://xunit.net/releases/visualstudio/2.5.2-pre.10

All that said? Please, still, fix this bug.

[nohwnd]

I've been meaning to respond here yesterday, but was too exhausted when I got to it in the evening.

The reason why I did not want to fix this is because Microsoft.NET.Test.Sdk was designed to do nothing on a .NET Framework project. The .NET Framework project test runner is shipped with vstest.console (e.g. in dotnet sdk, or VS). Adding a dependency to Test.SDK that is specifically for .NET Framework, would make the package even more confusing, especially when on Windows ObjectModel is picked up from the vstest.console folder.

For me the ideal fix would be to make the Test.SDK package work the same for .net framework as it works for .net, so it would ship testhost and the dependencies into the bin folder, and pick it up from there.

For the problem, I can see it reproduces with xunit when appdomains are enabled, but not when they are disabled. (-- RunConfiguration.DisableAppDomain=true). In mstest we register custom dll resolver in appdomain to pick up dlls from testhost folder when using appdomains. https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTestAdapter.PlatformServices/AssemblyResolver.cs#L408

[bradwilson]

@nohwnd Why does this work on Windows and not Linux? Are you secretly hooking something on behalf of the adapter in a way that doesn't work on Mono?

[nohwnd]

I don't know. I've seen mono being more picky about the dlls that .net framework, but I cannot pinpoint it. I've tried running the same thing on linux (wsl) through vstest.console.exe to make sure we have the dlls in all the same places, and I get the same error as when using dotnet test. There are no fusion logs as far as I know on mono, so I am not sure how to progress.

[bradwilson]

@nohwnd Can you point where in the code you determine the location of the testhost assembly? The link you pointed out just shows the resolution logic, but not how you constructed the list of folders to resolve with.

[nohwnd]

https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTestAdapter.PlatformServices/Services/TestSourceHost.cs#L357 It happens here, when the resolver is setup. It adds the directory of the host.

[bradwilson]

I shipped 2.5.3 yesterday with the NuGet package dependency against Microsoft.TestPlatform.ObjectModel and we'll see how users get along with that. I suspect the number of users trying to run xUnit v2 tests on Mono (which is not officially supported) is relatively low, and probably the majority of them had already fixed this with a manual package reference.

If that blows up in unexpected ways, I'll consider pulling it back out and replacing it with this assembly resolver. We already do assembly resolution for the test adapter path, which I suspect was/is only necessary inside Visual Studio (as dotnet test and friends currently depend on us copying the test adapter into the build output folder).