Support net452 tests in osx/linux (with mono)

[codito]

It is possible to build projects targeting net4x with dotnet build (using these steps: https://github.com/dotnet/netcorecli-fsc/wiki/.NET-Core-SDK-1.0.1#using-net-framework-as-targets-framework-the-osxunix-build-fails).

dotnet test should invoke testhost with mono runtime for net4x targets in osx/unix.

[codito]

Experimented a bit on an Ubuntu 16.04 box: Mono JIT compiler version 4.8.0 (Stable 4.8.0.520/8f6d0f6 Wed Mar 15 16:18:35 UTC 2017).

Tried to workaround by making sdk/TestHost/testhost.x86.exe a shell script and invoking the actual testhost.x86.exe with mono inside that. Here's the shell script I used:

#!/bin/bash                                                                                                                                                                                                                         
ppid=$$                                                                                                                                                                                                                             
command=$(echo "$@" | sed -e "s/--parentprocessid\s\d\s/--parentprocessid\s$ppid\s/")                                                                                                                                               
echo $command                                                                                                                                                                                                                       
mono ~/src/vstest/tools/dotnet/sdk/2.0.0-preview1-005685/TestHost/testhost2.x86.exe $command  

It seems testhost is exiting early and that breaks the communication pipe. The Process.Exited event on monitored parent process fires even if the vstest.console.dll process is still running. Note the timestamps in logs below. Error message from log.host.*.txt:

TpTrace Information: 0 : 15678, 1, 2017/04/01, 16:36:08.946, 2701610357608, testhost2.x86.exe, DefaultEngineInvoker: Monitoring parent process with id: '15659'
TpTrace Information: 0 : 15678, 1, 2017/04/01, 16:36:08.951, 2701610403579, testhost2.x86.exe, DefaultEngineInvoker: Initialize communication on port number: '34136'
TpTrace Information: 0 : 15678, 6, 2017/04/01, 16:36:08.953, 2701610426015, testhost2.x86.exe, DefaultEngineInvoker: ParentProcess '15659' Exited.
TpTrace Information: 0 : 15678, 1, 2017/04/01, 16:36:08.953, 2701610427552, testhost2.x86.exe, Trying to connect to server on port : 34136
TpTrace Information: 0 : 15678, 4, 2017/04/01, 16:36:08.966, 2701610559726, testhost2.x86.exe, Connected to the server successfully 
TpTrace Information: 0 : 15678, 1, 2017/04/01, 16:36:09.081, 2701611711252, testhost2.x86.exe, DefaultEngineInvoker: Start Request Processing.

Here's the messages from vstest.console at the same time log.txt:

TpTrace Verbose: 0 : 15659, 1, 2017/04/01, 16:36:09.080, 270161169353019, vstest.console.dll, TestRunRequest.ExecuteAsync: Wait for the first run request is over.
TpTrace Information: 0 : 15659, 1, 2017/04/01, 16:36:09.111, 270161201262796, vstest.console.dll, AssemblyResolver: System.Net.Sockets.resources: Resolving assembly.
TpTrace Information: 0 : 15659, 1, 2017/04/01, 16:36:09.112, 270161201839449, vstest.console.dll, AssemblyResolver: System.Net.Sockets.resources: Resolving assembly.
TpTrace Error: 0 : 15659, 1, 2017/04/01, 16:36:09.126, 270161215565998, vstest.console.dll, Server: TestExecution: Aborting test run because System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.WriteAndFlushToChannel(String rawMessage)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.SendMessage(String messageType, Object payload)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.StartTestRunAndListenAndReportTestResults(String messageType, Object payload, ITestRunEventsHandler eventHandler)
TpTrace Information: 0 : 15659, 1, 2017/04/01, 16:36:09.126, 270161215925283, vstest.console.dll, AssemblyResolver: Microsoft.TestPlatform.CommunicationUtilities.resources: Resolving assembly.

Need help to figure if this is a bug in mono.

[DavidKarlas]

Steps I used to run test:
1st apply this commit: DavidKarlas@ab09cdb

Restore+build

msbuild /t:Restore
msbuild

then copy TestHostRuntimeProvider into Extensions folder:

mkdir src/vstest.console/bin/Debug/net46/win7-x64/Extensions/
cp src/Microsoft.TestPlatform.TestHostProvider/bin/Debug/net46/Microsoft.TestPlatform.TestHostRuntimeProvider.* src/vstest.console/bin/Debug/net46/win7-x64/Extensions/

also copy TestHost

mkdir src/vstest.console/bin/Debug/net46/win7-x64/TestHost/
cp src/testhost.x86/bin/Debug/net46/win7-x64/* src/vstest.console/bin/Debug/net46/win7-x64/TestHost/

and then execute

mono src/vstest.console/bin/Debug/net46/win7-x64/vstest.console.exe --TestAdapterPath:test/Microsoft.TestPlatform.Common.UnitTests/bin/Debug/net46/ test/Microsoft.TestPlatform.Common.UnitTests/bin/Debug/net46/Microsoft.TestPlatform.Common.UnitTests.dll

I think you will know how to automate whole thing to load MSTest adapter without parameter and without all copying to Extensions and TestHost...

[codito]

Thanks @DavidKarlas, I have raised a PR for this (#851).

Is it expected that exe format will be always natively registered in Linux/Mac for mono? I'm wondering if we need to respect the launcher application here, i.e. (use mono to launch testhost.exe if user has launched mono vstest.console.exe).

[DavidKarlas]

Mono implementation of Process.Start(); on *nix(https://github.com/mono/mono/blob/8df83179b63e256bcd476d6d7ddbbd50015b81d6/mono/metadata/w32process-unix.c#L1820-L1823) check if executable is .NET assembly and starts with mono.

[codito]

@DavidKarlas I've merged #851. It unblocks running tests with a workaround. We're hitting an assembly load exception when the test adapters are loaded in testhost.x86.exe, this doesn't occur if we disable appdomains for the test run.

Steps that work after #851 is merged

# cd to vstest repo
> ./build.sh
> mono artifacts/Debug/net46/ubuntu.16.04-x64/vstest.console.exe samples/UnitTestProject/bin/Debug/net46/UnitTestProject.dll --framework:Framework45 --testAdapterPath:packages/mstest.testadapter/1.1.14/build/_common -- RunConfiguration.DisableAppDomain=true

Steps that fails due to assembly load

# cd to vstest repo
> ./build.sh
> mono artifacts/Debug/net46/ubuntu.16.04-x64/vstest.console.exe samples/UnitTestProject/bin/Debug/net46/UnitTestProject.dll --framework:Framework45 --testAdapterPath:packages/mstest.testadapter/1.1.14/build/_common

If DisableAppDomain=false, the test adapters in testhost.x86.exe process, will try to create an AppDomain to run the unit tests. Inside the AppDomain, mono is unable to find a testplatform assembly (there isn't any code specific to mono in this path, it works in .NET full CLR). Logs with MONO_LOG_LEVEL=debug are below. 11.0.0.0 version of ObjectModel is redirected to 15.0.0.0 in assembly bindings in testhost.x86.exe.config, it successfully loads in default appdomain, but fails for some reason in child appdomain.

Debug log

Mono: Assembly System.Globalization[0x7f2b90095a20] added to domain TestSourceHost: Enumering assembly, ref_count=3                                                                                                                                                                   
12715 Mono: AOT: image '/usr/lib/mono/4.5/Facades/System.Globalization.dll.so' not found: /usr/lib/mono/4.5/Facades/System.Globalization.dll.so: cannot open shared object file: No such file or directory                                                                                  
12716 Mono: AOT: image '/usr/lib/mono/aot-cache/amd64/System.Globalization.dll.so' not found: /usr/lib/mono/aot-cache/amd64/System.Globalization.dll.so: cannot open shared object file: No such file or directory                                                                          
12717 Mono: Assembly Ref addref Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter[0x7f2b90864290] -> System.Globalization[0x7f2b90095a20]: 4                                                                                                                                           
12718 Mono: Config attempting to parse: '/home/armahapa/src/vstest/artifacts/Debug/net46/ubuntu.16.04-x64/TestHost/testhost.x86.exe.config'.                                                                                                                                                
12719 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                           
12720 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                                               
12721 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                             
12722 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                           
12723 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                                               
12724 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                             
12725 Mono: Found assembly remapping for System.Core and was for the same version 4.0.0.0                                                                                                                                                                                                   
12726 Mono: Found assembly remapping for System.Core and was for the same version 4.0.0.0                                                                                                                                                                                                   
12727 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll'.                                                                                                                                                                    
12728 Mono: Assembly System.Core[0x7f2b8801d730] added to domain TestSourceHost: Enumering assembly, ref_count=9                                                                                                                                                                            
12729 Mono: AOT: image '/usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll.so' not found: /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll.so: cannot open shared object file: No such file or directory                                        
12730 Mono: AOT: image '/usr/lib/mono/aot-cache/amd64/System.Core.dll.so' not found: /usr/lib/mono/aot-cache/amd64/System.Core.dll.so: cannot open shared object file: No such file or directory                                                                                            
12731 Mono: Assembly Ref addref Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices[0x7f2b9076f900] -> System.Core[0x7f2b8801d730]: 10                                                                                                                                       
12732 Mono: Unloading image /home/armahapa/src/vstest/artifacts/Debug/net46/ubuntu.16.04-x64/TestHost/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll [0x7f2b905abaf0].                                                                                                                 
12733 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                           
12734 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                                               
12735 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                             
12736 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                           
12737 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                                               
12738 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                             
12739 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                           
12740 Mono: Assembly Loader probing location: '/home/armahapa/src/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                      
12741 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                                               
12742 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'.                                                                                                                                                             
12743 Mono: Assembly Loader probing location: '/usr/lib/mono/gac/Microsoft.VisualStudio.TestPlatform.ObjectModel/15.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                           
12744 Mono: Assembly Loader probing location: '/home/armahapa/src/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                      
12745 Mono: Assembly Loader probing location: '/usr/lib/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                                               
12746 Mono: Assembly Loader probing location: '/usr/lib/mono/4.5//Facades/Microsoft.VisualStudio.TestPlatform.ObjectModel.exe'.                                                                                                                                                             
12747 Mono: The following assembly referenced from /home/armahapa/src/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll could not be loaded:                                                                                        
12748      Assembly:   Microsoft.VisualStudio.TestPlatform.ObjectModel    (assemblyref_index=3)                                                                                                                                                                                             
12749      Version:    11.0.0.0                                                                                                                                                                                                                                                             
12750      Public Key: b03f5f7f11d50a3a                                                                                                                                                                                                                                                     
12751 The assembly was not found in the Global Assembly Cache, a path listed in the MONO_PATH environment variable, or in the location of the executing assembly (/home/armahapa/src/vstest/samples/UnitTestProject/bin/Debug/net46/).                                                      

[DavidKarlas]

I'm reproducing kinda same problem on Windows:

*** Assembly Binder Log Entry  (6/21/2017 @ 12:50:44 PM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Users\vs2017\GIT\vstest\artifacts\Debug\net46\win7-x64\TestHost\testhost.x86.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = NULL
Calling assembly : Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Users\vs2017\GIT\vstest\artifacts\Debug\net46\win7-x64\TestHost\testhost.x86.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Redirect found in application configuration file: 11.0.0.0 redirected to 15.0.0.0.
LOG: Post-policy reference: Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel.DLL.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel/Microsoft.VisualStudio.TestPlatform.ObjectModel.DLL.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Extensions/Microsoft.VisualStudio.TestPlatform.ObjectModel.DLL.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Extensions/Microsoft.VisualStudio.TestPlatform.ObjectModel/Microsoft.VisualStudio.TestPlatform.ObjectModel.DLL.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel.EXE.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Microsoft.VisualStudio.TestPlatform.ObjectModel/Microsoft.VisualStudio.TestPlatform.ObjectModel.EXE.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Extensions/Microsoft.VisualStudio.TestPlatform.ObjectModel.EXE.
LOG: Attempting download of new URL file:///C:/Users/vs2017/GIT/vstest/samples/UnitTestProject/bin/Debug/net46/Extensions/Microsoft.VisualStudio.TestPlatform.ObjectModel/Microsoft.VisualStudio.TestPlatform.ObjectModel.EXE.
LOG: All probing URLs attempted and failed.

Only difference is that Mono crashes, and Windows for some reason continues execution

[Faizan2304]

@DavidKarlas : Please try with latest dotnet tools if you are still facing this issue. Closing this issue for now. Please reopen it if you have any thing to discuss here.