Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Differences in TestMessage log output between xunit and mstest #1222

Closed
alistair opened this issue Jun 11, 2018 · 16 comments
Closed

Differences in TestMessage log output between xunit and mstest #1222

alistair opened this issue Jun 11, 2018 · 16 comments

Comments

@alistair
Copy link

Im implementing omnisharp-emacs:Run all tests in class and had planned to output the test logs into the compile buffer. ( 2 get go to file etc support ).

When running against xunit i get the following output

Microsoft (R) Build Engine version 15.5.180.51428 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  TestProject -> /Users/alistair.bush/projects/TestProject/bin/Debug/netcoreapp2.0/TestProject.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.74


[xUnit.net 00:00:00.4129590]   Discovering: TestProject
[xUnit.net 00:00:00.4875360]   Discovered:  TestProject
[xUnit.net 00:00:00.4336950]   Starting:    TestProject
[xUnit.net 00:00:00.6051810]     TestProject.UnitTest1.Test1 [FAIL]
[xUnit.net 00:00:00.6063010]       Assert.True() Failure
[xUnit.net 00:00:00.6063630]       Expected: True
[xUnit.net 00:00:00.6063890]       Actual:   False
[xUnit.net 00:00:00.6074430]       Stack Trace:
[xUnit.net 00:00:00.6089270]         /Users/alistair.bush/projects/TestProject/UnitTest1.cs(11,0): at TestProject.UnitTest1.Test1()
[xUnit.net 00:00:00.6210890]   Finished:    TestProject

but it seems like mstest doesn't output these logs. and therefore I get the following output (which is just the compile output).

Microsoft (R) Build Engine version 15.5.180.51428 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  MSTestProject -> /Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.79

Im suspecting this is a due to vstest for mstest not outputing any console logs. Would this be correct?

Also would it be possible to affect this by passing /logger:console?

@DustinCampbell
Copy link
Contributor

cc @akshita31

@akshita31
Copy link
Contributor

@alistair I tried this sample mstest project and am getting the required logs for the mstest project system. Can you try this project?

Also, can you share the project where you are not getting the logs and also the output of dotnet --info for your machine?

@akshita31
Copy link
Contributor

FWIW, I was running into the same problem for this project : https://github.com/OmniSharp/omnisharp-roslyn/tree/master/test-assets/test-projects/MSTestProject. Then on checking the logs from OmniSharp I noticed the following:

'\omnisharp-roslyn\test-assets\test-projects\MSTestProject\obj\project.assets.json' doesn't have a target for '.NETCoreApp,Version=v1.1'. Ensure that restore has run and that you have included 'netcoreapp1.1' in the TargetFrameworks for your project

Then on doing so and performing the restore, I am able to get the desired output. Can you check if this is the same case for you?

@alistair
Copy link
Author

alistair commented Jul 2, 2018

Hi @akshita31 I was using just the template mstest project for .net sdk 2.0

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.2.0" />
    <PackageReference Include="MSTest.TestFramework" Version="1.2.0" />
  </ItemGroup>

</Project>
% dotnet --info                                                                                                                                                                                                     !5006
.NET Command Line Tools (2.1.4)

Product Information:
 Version:            2.1.4
 Commit SHA-1 hash:  5e8add2190

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.1.4/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.6
  Build    : 74b1c703813c8910df5b96f304b0f2b78cdf194d

@alistair
Copy link
Author

alistair commented Jul 2, 2018

This is the output if I run omnisharp from the command line and post in the command to run the tests.

% ~/.emacs.d/.cache/omnisharp/server/v1.30.1/run                                                                                                                                                                    !5011
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Stdio.Host","Message":"Starting OmniSharp on MacOS 10.13.5 (x64)"},"Seq":1,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.Discovery.MSBuildLocator","Message":"Located 1 MSBuild instance(s)\n            1: StandAlone 15.0 - \"/Users/alistair.bush/.emacs.d/.cache/omnisharp/server/v1.30.1/omnisharp/msbuild/15.0/Bin\""},"Seq":2,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.Discovery.MSBuildLocator","Message":"MSBUILD_EXE_PATH environment variable set to '/Users/alistair.bush/.emacs.d/.cache/omnisharp/server/v1.30.1/omnisharp/msbuild/15.0/Bin/MSBuild.dll'"},"Seq":3,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.Discovery.MSBuildLocator","Message":"Registered MSBuild instance: StandAlone 15.0 - \"/Users/alistair.bush/.emacs.d/.cache/omnisharp/server/v1.30.1/omnisharp/msbuild/15.0/Bin\"\n            MSBuildExtensionsPath = /Library/Frameworks/Mono.framework/Versions/5.2.0/lib/mono/xbuild\n            BypassFrameworkInstallChecks = true\n            CscToolPath = /Users/alistair.bush/.emacs.d/.cache/omnisharp/server/v1.30.1/omnisharp/msbuild/15.0/Bin/Roslyn\n            CscToolExe = csc.exe\n            MSBuildToolsPath = /Users/alistair.bush/.emacs.d/.cache/omnisharp/server/v1.30.1/omnisharp/msbuild/15.0/Bin\n            TargetFrameworkRootPath = /Library/Frameworks/Mono.framework/Versions/5.2.0/lib/mono/xbuild-frameworks"},"Seq":4,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Cake.CakeProjectSystem","Message":"Detecting Cake files in '/Users/alistair.bush/projects/MSTestProject'."},"Seq":5,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Cake.CakeProjectSystem","Message":"Could not find any Cake files"},"Seq":6,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNet.DotNetProjectSystem","Message":"Initializing in /Users/alistair.bush/projects/MSTestProject"},"Seq":7,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNet.DotNetProjectSystem","Message":"Auto package restore: False"},"Seq":8,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNet.DotNetProjectSystem","Message":"Update workspace context"},"Seq":9,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNet.DotNetProjectSystem","Message":"Resolving projects references"},"Seq":10,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.ProjectSystem","Message":"Detecting projects in '/Users/alistair.bush/projects/MSTestProject/MSTestProject.sln'."},"Seq":11,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.ProjectManager","Message":"Queue project update for '/Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj'"},"Seq":12,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Script.ScriptProjectSystem","Message":"Detecting CSX files in '/Users/alistair.bush/projects/MSTestProject'."},"Seq":13,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Script.ScriptProjectSystem","Message":"Could not find any CSX files"},"Seq":14,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Stdio.Host","Message":"Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpWorkspaceOptionsProvider"},"Seq":15,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Stdio.Host","Message":"Configuration finished."},"Seq":16,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.Stdio.Host","Message":"Omnisharp server running using Stdio at location '/Users/alistair.bush/projects/MSTestProject' on host -1."},"Seq":17,"Type":"event"}
{"Event":"started","Body":null,"Seq":18,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.ProjectManager","Message":"Loading project: /Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj"},"Seq":19,"Type":"event"}
{"Event":"MsBuildProjectDiagnostics","Body":{"FileName":"/Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj","Warnings":[],"Errors":[]},"Seq":20,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.ProjectManager","Message":"Adding project '/Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj'"},"Seq":21,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.MSBuild.ProjectManager","Message":"Update project: MSTestProject"},"Seq":22,"Type":"event"}
{"Event":"ProjectAdded","Body":{"MsBuildProject":{"ProjectGuid":"00000000-0000-0000-0000-000000000000","Path":"/Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj","AssemblyName":"MSTestProject","TargetPath":"/Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll","TargetFramework":".NETCoreApp,Version=v2.0","SourceFiles":["/Users/alistair.bush/projects/MSTestProject/UnitTest1.cs","/var/folders/ld/xcrpj7dd43gghw4zkgzfpz94dqnl2t/T/.NETCoreApp,Version=v2.0.AssemblyAttributes.cs","/Users/alistair.bush/projects/MSTestProject/obj/Debug/netcoreapp2.0/MSTestProject.Program.cs","/Users/alistair.bush/projects/MSTestProject/obj/Debug/netcoreapp2.0/MSTestProject.AssemblyInfo.cs"],"TargetFrameworks":[{"Name":".NETCoreApp","FriendlyName":".NETCoreApp","ShortName":"netcoreapp2.0"}],"OutputPath":"bin/Debug/netcoreapp2.0/","IsExe":true,"IsUnityProject":false}},"Seq":23,"Type":"event"}
{"Event":"ProjectChanged","Body":{"MsBuildProject":{"ProjectGuid":"00000000-0000-0000-0000-000000000000","Path":"/Users/alistair.bush/projects/MSTestProject/MSTestProject.csproj","AssemblyName":"MSTestProject","TargetPath":"/Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll","TargetFramework":".NETCoreApp,Version=v2.0","SourceFiles":["/Users/alistair.bush/projects/MSTestProject/UnitTest1.cs","/var/folders/ld/xcrpj7dd43gghw4zkgzfpz94dqnl2t/T/.NETCoreApp,Version=v2.0.AssemblyAttributes.cs","/Users/alistair.bush/projects/MSTestProject/obj/Debug/netcoreapp2.0/MSTestProject.Program.cs","/Users/alistair.bush/projects/MSTestProject/obj/Debug/netcoreapp2.0/MSTestProject.AssemblyInfo.cs"],"TargetFrameworks":[{"Name":".NETCoreApp","FriendlyName":".NETCoreApp","ShortName":"netcoreapp2.0"}],"OutputPath":"bin/Debug/netcoreapp2.0/","IsExe":true,"IsUnityProject":false}},"Seq":24,"Type":"event"}
{"Type":"request", "Seq": 11, "Command":"/v2/runtestsinclass", "Arguments": { "FileName":"/Users/alistair.bush/projects/MSTestProject/UnitTest1.cs", "MethodNames": ["MSTestProject.UnitTest1.TestMethod1", "MSTestProject.UnitTest1.TestMethod2"], "TestFrameworkName": "mstest", "TargetFrameworkVersion": ".NETCoreApp, Version=2.0" }}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"Microsoft (R) Build Engine version 15.5.180.51428 for .NET Core"},"Seq":26,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"Copyright (C) Microsoft Corporation. All rights reserved."},"Seq":27,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":""},"Seq":28,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"  MSTestProject -> /Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll"},"Seq":29,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":""},"Seq":30,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"Build succeeded."},"Seq":31,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"    0 Warning(s)"},"Seq":32,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"    0 Error(s)"},"Seq":33,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":""},"Seq":34,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":"Time Elapsed 00:00:01.83"},"Seq":35,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"informational","Message":""},"Seq":36,"Type":"event"}
{"Event":"TestMessage","Body":{"MessageLevel":"error","Message":""},"Seq":37,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNetTest.VSTestManager","Message":"read: {\"MessageType\":\"TestSession.Connected\",\"Payload\":null}"},"Seq":38,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNetTest.VSTestManager","Message":"read: {\"MessageType\":\"ProtocolVersion\",\"Payload\":1}"},"Seq":39,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNetTest.VSTestManager","Message":"read: {\"MessageType\":\"TestExecution.StatsChange\",\"Payload\":{\"NewTestResults\":[{\"TestCase\":{\"Properties\":[{\"Key\":{\"Id\":\"TestCase.FullyQualifiedName\",\"Label\":\"FullyQualifiedName\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":\"MSTestProject.UnitTest1.TestMethod1\"},{\"Key\":{\"Id\":\"TestCase.ExecutorUri\",\"Label\":\"Executor Uri\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Uri\"},\"Value\":\"executor://MSTestAdapter/v2\"},{\"Key\":{\"Id\":\"TestCase.Source\",\"Label\":\"Source\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"/Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll\"},{\"Key\":{\"Id\":\"TestCase.CodeFilePath\",\"Label\":\"File Path\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"/Users/alistair.bush/projects/MSTestProject/UnitTest1.cs\"},{\"Key\":{\"Id\":\"TestCase.DisplayName\",\"Label\":\"Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"TestMethod1\"},{\"Key\":{\"Id\":\"TestCase.Id\",\"Label\":\"Id\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Guid\"},\"Value\":\"12713494-ea68-505c-1ae3-a995253433bd\"},{\"Key\":{\"Id\":\"TestCase.LineNumber\",\"Label\":\"Line Number\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Int32\"},\"Value\":10},{\"Key\":{\"Id\":\"MSTestDiscoverer.TestClassName\",\"Label\":\"ClassName\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":\"MSTestProject.UnitTest1\"},{\"Key\":{\"Id\":\"TestObject.Traits\",\"Label\":\"Traits\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":5,\"ValueType\":\"System.Collections.Generic.KeyValuePair`2[[System.String],[System.String]][]\"},\"Value\":[]}]},\"Attachments\":[],\"Messages\":[],\"Properties\":[{\"Key\":{\"Id\":\"TestResult.Outcome\",\"Label\":\"Outcome\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"Microsoft.VisualStudio.TestPlatform.ObjectModel.TestOutcome, Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},\"Value\":2},{\"Key\":{\"Id\":\"TestResult.ErrorMessage\",\"Label\":\"Error Message\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"Assert.AreEqual failed. Expected:<1>. Actual:<2>. \"},{\"Key\":{\"Id\":\"TestResult.ErrorStackTrace\",\"Label\":\"Error Stack Trace\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"   at MSTestProject.UnitTest1.TestMethod1() in /Users/alistair.bush/projects/MSTestProject/UnitTest1.cs:line 11\\n\"},{\"Key\":{\"Id\":\"TestResult.DisplayName\",\"Label\":\"TestResult Display Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":null},{\"Key\":{\"Id\":\"TestResult.ComputerName\",\"Label\":\"Computer Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"\"},{\"Key\":{\"Id\":\"TestResult.Duration\",\"Label\":\"Duration\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.TimeSpan\"},\"Value\":\"00:00:00.0265000\"},{\"Key\":{\"Id\":\"TestResult.StartTime\",\"Label\":\"Start Time\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.DateTimeOffset\"},\"Value\":\"2018-07-02T22:34:49.007236+12:00\"},{\"Key\":{\"Id\":\"TestResult.EndTime\",\"Label\":\"End Time\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.DateTimeOffset\"},\"Value\":\"2018-07-02T22:34:49.073944+12:00\"}]},{\"TestCase\":{\"Properties\":[{\"Key\":{\"Id\":\"TestCase.FullyQualifiedName\",\"Label\":\"FullyQualifiedName\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":\"MSTestProject.UnitTest1.TestMethod2\"},{\"Key\":{\"Id\":\"TestCase.ExecutorUri\",\"Label\":\"Executor Uri\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Uri\"},\"Value\":\"executor://MSTestAdapter/v2\"},{\"Key\":{\"Id\":\"TestCase.Source\",\"Label\":\"Source\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"/Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll\"},{\"Key\":{\"Id\":\"TestCase.CodeFilePath\",\"Label\":\"File Path\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"/Users/alistair.bush/projects/MSTestProject/UnitTest1.cs\"},{\"Key\":{\"Id\":\"TestCase.DisplayName\",\"Label\":\"Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"TestMethod2\"},{\"Key\":{\"Id\":\"TestCase.Id\",\"Label\":\"Id\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Guid\"},\"Value\":\"86f66277-b785-32be-2728-6afa4772ed4f\"},{\"Key\":{\"Id\":\"TestCase.LineNumber\",\"Label\":\"Line Number\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.Int32\"},\"Value\":16},{\"Key\":{\"Id\":\"MSTestDiscoverer.TestClassName\",\"Label\":\"ClassName\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":\"MSTestProject.UnitTest1\"},{\"Key\":{\"Id\":\"TestObject.Traits\",\"Label\":\"Traits\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":5,\"ValueType\":\"System.Collections.Generic.KeyValuePair`2[[System.String],[System.String]][]\"},\"Value\":[]}]},\"Attachments\":[],\"Messages\":[],\"Properties\":[{\"Key\":{\"Id\":\"TestResult.Outcome\",\"Label\":\"Outcome\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"Microsoft.VisualStudio.TestPlatform.ObjectModel.TestOutcome, Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"},\"Value\":1},{\"Key\":{\"Id\":\"TestResult.ErrorMessage\",\"Label\":\"Error Message\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":null},{\"Key\":{\"Id\":\"TestResult.ErrorStackTrace\",\"Label\":\"Error Stack Trace\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":null},{\"Key\":{\"Id\":\"TestResult.DisplayName\",\"Label\":\"TestResult Display Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":1,\"ValueType\":\"System.String\"},\"Value\":null},{\"Key\":{\"Id\":\"TestResult.ComputerName\",\"Label\":\"Computer Name\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.String\"},\"Value\":\"\"},{\"Key\":{\"Id\":\"TestResult.Duration\",\"Label\":\"Duration\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.TimeSpan\"},\"Value\":\"00:00:00.0000870\"},{\"Key\":{\"Id\":\"TestResult.StartTime\",\"Label\":\"Start Time\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.DateTimeOffset\"},\"Value\":\"2018-07-02T22:34:49.082873+12:00\"},{\"Key\":{\"Id\":\"TestResult.EndTime\",\"Label\":\"End Time\",\"Category\":\"\",\"Description\":\"\",\"Attributes\":0,\"ValueType\":\"System.DateTimeOffset\"},\"Value\":\"2018-07-02T22:34:49.083147+12:00\"}]}],\"TestRunStatistics\":{\"ExecutedTests\":2,\"Stats\":{\"Failed\":1,\"Passed\":1}},\"ActiveTests\":[]}}"},"Seq":40,"Type":"event"}
{"Event":"log","Body":{"LogLevel":"INFORMATION","Name":"OmniSharp.DotNetTest.VSTestManager","Message":"read: {\"MessageType\":\"TestExecution.Completed\",\"Payload\":{\"TestRunCompleteArgs\":{\"TestRunStatistics\":{\"ExecutedTests\":2,\"Stats\":{\"Failed\":1,\"Passed\":1}},\"IsCanceled\":false,\"IsAborted\":false,\"Error\":null,\"AttachmentSets\":[],\"ElapsedTimeInRunningTests\":\"00:00:00.1040000\",\"Metrics\":{}},\"LastRunTests\":null,\"RunAttachments\":[],\"ExecutorUris\":[\"executor://mstestadapter/v2\"]}}"},"Seq":41,"Type":"event"}
{"Request_seq":11,"Command":"/v2/runtestsinclass","Running":true,"Success":true,"Message":null,"Body":{"Results":[{"MethodName":"MSTestProject.UnitTest1.TestMethod1","Outcome":"failed","ErrorMessage":"Assert.AreEqual failed. Expected:<1>. Actual:<2>. ","ErrorStackTrace":"   at MSTestProject.UnitTest1.TestMethod1() in /Users/alistair.bush/projects/MSTestProject/UnitTest1.cs:line 11\n"},{"MethodName":"MSTestProject.UnitTest1.TestMethod2","Outcome":"passed","ErrorMessage":null,"ErrorStackTrace":null}],"Pass":false,"Failure":null},"Seq":25,"Type":"response"}

It seems like the compile part outputs TestMessages but once the tests run we don't get any actual console output from the mstest. This is something which xunit provides.

@alistair
Copy link
Author

alistair commented Jul 2, 2018

BTW here is the actual test class

namespace MSTestProject
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
          Assert.AreEqual(1, 2);
        }

        [TestMethod]
        public void TestMethod2()
        {
          Assert.AreEqual(1, 1);
        }
    }
}

@alistair
Copy link
Author

alistair commented Jul 2, 2018

I probably haven't explained what im after very well. Essentially I would like the console log output as if I had run dotnet test myself. It seems that xunit already outputs this from omnisharp in the form of TestMessage messages.

for example the output from dotnet test from the mstest project.

Build started, please wait...
Build completed.

Test run for /Users/alistair.bush/projects/MSTestProject/bin/Debug/netcoreapp2.0/MSTestProject.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Failed   TestMethod1
Error Message:
 Assert.AreEqual failed. Expected:<1>. Actual:<2>.
Stack Trace:
   at MSTestProject.UnitTest1.TestMethod1() in /Users/alistair.bush/projects/MSTestProject/UnitTest1.cs:line 11


Total tests: 2. Passed: 1. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.1724 Seconds

@alistair
Copy link
Author

alistair commented Jul 2, 2018

@akshita31 I tried the other project you linked but im sorry I don't think I understand. I can't see that log message you provide and don't understand why it would be there anyway. These are netcoreapp2.0 test projects, why would they need netcoreapp1.1?

Also it seems like the test execute as expected, but nothing is console logs for them.

@akshita31
Copy link
Contributor

@alistair My apologies, I could not understand the problem previously and yes I can totally reproduce it now. I am investigating further into why is the difference coming up.

@akshita31
Copy link
Contributor

To provide a more solid repro:
Project: https://github.com/xunit/samples.xunit/tree/master/AssertExamples
Executing a RunTest request for any of the above, gives some console logs, but there is no DotNetTestResult returned (the part in the image after Test Execution Summary)
xunit

Whereas for MSTest project:
Project: https://github.com/dotnet/samples/tree/master/core/getting-started/unit-testing-using-mstest
There are no console logs after the build stage but we get a DotNetTestResult response:
mstest

(Nit: I am using the vscode extension for the above screenshots)

@akshita31
Copy link
Contributor

akshita31 commented Jul 2, 2018

@DustinCampbell @rchande I checked in omnisharp and the TestManager has nothing that should imply different behaviors for xunit and mstest system. I think it is the vstest that is giving different message formats for them. in Xunit there is just a TestMessage type and the ExecutionComplete type. In Mstest there is TestRunStatsChange type which includes the test results. Could any one of you verify this ?

Here is the code in question :

var message = ReadMessage();

@alistair
Copy link
Author

alistair commented Jul 3, 2018

One should also probably ask if i can/should rely on the test output being available.

The reason im relying on it is because it seems like a easy and effective way of supporting this within editors without codelens or other ways of displaying inline test statuses.

Emacs for example has a compilation buffer functionality which i can use and which already supports goto file and line for a number of different output types. Including msbuild xbuild and other langs.

@akshita31
Copy link
Contributor

akshita31 commented Aug 2, 2018

@DustinCampbell Can you help with this one ?

@DustinCampbell
Copy link
Contributor

@akshita31 : What sort of help are you looking for?

@akshita31
Copy link
Contributor

@DustinCampbell Can you confirm if I am thinking in the right direction: #1222 (comment)

@DustinCampbell
Copy link
Contributor

Maybe? It'd probably be better to reach out to some folks on the vs test team. @AbhitejJohn is the engineer who originally implemented MSTest support in OmniSharp (#856) and he sits down the hallway from @rchande. He might have some insight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants