Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Cambridge test suite (tests/fsharp) runner to NUnit #90

Merged
merged 7 commits into from
Nov 16, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages.config
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@
<package id="NUnit" version="2.6.4" targetFramework="net40" />
<package id="NUnit.Runners" version="2.6.4" />
<package id="FsCheck" version="2.0.3" />
<package id="NUnit" version="3.0.0-beta-3" targetFramework="net45" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really want two different versions of NUnit? How about removing the old one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the old nunit is used in unit tests projects.
These proj are going to be converted to xunit instead of upgrade to nunit 3 in coreclr branch ( see #699 ) temporary or not.
Now nunit 3 final is release, so maybe is easier to upgrade the suite to nunit 3 final in all projects (usefull for coreclr too), but that's another pr

<package id="NUnit.Console" version="3.0.0-beta-3" targetFramework="net45" />
</packages>
11 changes: 11 additions & 0 deletions src/fsharp.sln
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FsiAnyCPU", "fsharp\fsiAnyCpu\FsiAnyCPU.fsproj", "{8B3E283D-B5FE-4055-9D80-7E3A32F3967B}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Compiler.Unittests", "fsharp\FSharp.Compiler.Unittests\FSharp.Compiler.Unittests.fsproj", "{A8D9641A-9170-4CF4-8FE0-6DB8C134E1B5}"
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Tests", "Tests", "{E49B77EA-64DE-451A-A5E6-A6632FC1F03C}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Tests", "..\tests\fsharp\FSharp.Tests.fsproj", "{C163E892-5BF7-4B59-AA99-B0E8079C67C4}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down Expand Up @@ -81,8 +84,16 @@ Global
{A8D9641A-9170-4CF4-8FE0-6DB8C134E1B5}.Debug|Any CPU.Build.0 = Debug|Any CPU
{A8D9641A-9170-4CF4-8FE0-6DB8C134E1B5}.Release|Any CPU.ActiveCfg = Release|Any CPU
{A8D9641A-9170-4CF4-8FE0-6DB8C134E1B5}.Release|Any CPU.Build.0 = Release|Any CPU
{C163E892-5BF7-4B59-AA99-B0E8079C67C4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C163E892-5BF7-4B59-AA99-B0E8079C67C4}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C163E892-5BF7-4B59-AA99-B0E8079C67C4}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C163E892-5BF7-4B59-AA99-B0E8079C67C4}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{88E2D422-6852-46E3-A740-83E391DC7973} = {E49B77EA-64DE-451A-A5E6-A6632FC1F03C}
{C163E892-5BF7-4B59-AA99-B0E8079C67C4} = {E49B77EA-64DE-451A-A5E6-A6632FC1F03C}
EndGlobalSection
EndGlobal
186 changes: 186 additions & 0 deletions src/update.fs
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@

module UpdateCmd
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment about what this file is for.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the 1-1 conversion from src/update.cmd.
Yuo are right, i'll add a comment with the original .bat file path in all converted .fs files, it's easier to understand.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I meant was "please take the time to record what this stuff is actually doing" :). I know the original cmd files weren't documented (and of course they should have been) - but now you're going through all this with such a fine toothcomb, it's a great opportunity to add a few basic comments recording what you've learned.


open System.IO
open NUnit.Framework
open Microsoft.Win32

open PlatformHelpers
open FSharpTestSuiteTypes

type Configuration =
| DEBUG
| RELEASE
override this.ToString() =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this - is %A printing used on this type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i use that for ConfigurationName argument of msbuild or some bat invocation.
Maybe is better closer to usage instead of generic .ToString().
I'll move this functions closer to usage

match this with
| DEBUG -> "Debug"
| RELEASE -> "Release"

type updateCmdArgs =
{ Configuration : Configuration
Ngen : bool }

let private regQuery = WindowsPlatform.regQuery

let private checkResult result =
match result with
| CmdResult.ErrorLevel err -> let x = err, (sprintf "ERRORLEVEL %d" err) in Failure (RunError.ProcessExecError x)
| CmdResult.Success -> Success ()

let updateCmd envVars args = processor {
// @echo off
// setlocal
ignore "useless"

// if /i "%1" == "debug" goto :ok
// if /i "%1" == "release" goto :ok
ignore "already validated input"

// echo GACs built binaries, adds required strong name verification skipping, and optionally NGens built binaries
// echo Usage:
// echo update.cmd debug [-ngen]
// echo update.cmd release [-ngen]
// exit /b 1
ignore "useless help"

//:ok
let env k () = match envVars |> Map.tryFind k with None -> Failure (sprintf "environment variable '%s' not found" k) | Some x -> Success x
let ``~dp0`` = __SOURCE_DIRECTORY__
let exec exe args =
log "%s %s" exe args
use toLog = redirectToLog ()
Process.exec { RedirectError = Some toLog.Post; RedirectOutput = Some toLog.Post; RedirectInput = None } ``~dp0`` envVars exe args

// set BINDIR=%~dp0..\%1\net40\bin
let! binDir = env "FSCBINPATH"

// if /i "%PROCESSOR_ARCHITECTURE%"=="x86" set X86_PROGRAMFILES=%ProgramFiles%
// if /I "%PROCESSOR_ARCHITECTURE%"=="AMD64" set X86_PROGRAMFILES=%ProgramFiles(x86)%
let processorArchitecture = WindowsPlatform.processorArchitecture envVars
let x86_ProgramFiles = WindowsPlatform.x86ProgramFilesDirectory envVars processorArchitecture

let! windir = env "windir"

let REGEXE32BIT path value =
let hklm32 = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry32)
match hklm32 |> regQuery path value with
| Some (:? string as d) -> Some d
| Some _ | None -> None

let allWINSDKNETFXTOOLS = seq {
// FOR /F "tokens=2* delims= " %%A IN ('%REGEXE32BIT% QUERY "HKLM\Software\Microsoft\Microsoft SDKs\NETFXSDK\4.6\WinSDK-NetFx40Tools" /v InstallationFolder') DO SET WINSDKNETFXTOOLS=%%B
yield REGEXE32BIT @"Software\Microsoft\Microsoft SDKs\NETFXSDK\4.6\WinSDK-NetFx40Tools" "InstallationFolder"
// if "%WINSDKNETFXTOOLS%"=="" FOR /F "tokens=2* delims= " %%A IN ('%REGEXE32BIT% QUERY "HKLM\Software\Microsoft\Microsoft SDKs\Windows\v8.1A\WinSDK-NetFx40Tools" /v InstallationFolder') DO SET WINSDKNETFXTOOLS=%%B
yield REGEXE32BIT @"Software\Microsoft\Microsoft SDKs\Windows\v8.1A\WinSDK-NetFx40Tools" "InstallationFolder"
// if "%WINSDKNETFXTOOLS%"=="" FOR /F "tokens=2* delims= " %%A IN ('%REGEXE32BIT% QUERY "HKLM\Software\Microsoft\Microsoft SDKs\Windows\v8.0A\WinSDK-NetFx40Tools" /v InstallationFolder') DO SET WINSDKNETFXTOOLS=%%B
yield REGEXE32BIT @"Software\Microsoft\Microsoft SDKs\Windows\v8.0A\WinSDK-NetFx40Tools" "InstallationFolder"
// if "%WINSDKNETFXTOOLS%"=="" FOR /F "tokens=2* delims= " %%A IN ('%REGEXE32BIT% QUERY "HKLM\Software\Microsoft\Microsoft SDKs\Windows\v7.1\WinSDK-NetFx40Tools" /v InstallationFolder') DO SET WINSDKNETFXTOOLS=%%B
yield REGEXE32BIT @"Software\Microsoft\Microsoft SDKs\Windows\v7.1\WinSDK-NetFx40Tools" "InstallationFolder"
// if "%WINSDKNETFXTOOLS%"=="" FOR /F "tokens=2* delims= " %%A IN ('%REGEXE32BIT% QUERY "HKLM\Software\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools" /v InstallationFolder') DO SET WINSDKNETFXTOOLS=%%B
yield REGEXE32BIT @"Software\Microsoft\Microsoft SDKs\Windows\v7.0A\WinSDK-NetFx40Tools" "InstallationFolder"
}

let WINSDKNETFXTOOLS = match allWINSDKNETFXTOOLS |> Seq.tryPick id with Some sdk -> sdk | None -> ""

// set GACUTIL="%WINSDKNETFXTOOLS%gacutil.exe"
let GACUTIL = WINSDKNETFXTOOLS/"gacutil.exe"
// set SN32="%WINSDKNETFXTOOLS%sn.exe"
let SN32 = WINSDKNETFXTOOLS/"sn.exe"
// set SN64="%WINSDKNETFXTOOLS%x64\sn.exe"
let SN64 = WINSDKNETFXTOOLS/"x64"/"sn.exe"
// set NGEN32=%windir%\Microsoft.NET\Framework\v4.0.30319\ngen.exe
let NGEN32 = windir/"Microsoft.NET"/"Framework"/"v4.0.30319"/"ngen.exe"
// set NGEN64=%windir%\Microsoft.NET\Framework64\v4.0.30319\ngen.exe
let NGEN64 = windir/"Microsoft.NET"/"Framework64"/"v4.0.30319"/"ngen.exe"

let checkResult = function CmdResult.ErrorLevel err -> Failure (sprintf "ERRORLEVEL %d" err) | CmdResult.Success -> Success ()

let gacutil flags = Commands.gacutil exec GACUTIL flags >> checkResult
let ngen32 = Commands.ngen exec NGEN32 >> checkResult
let ngen64 = Commands.ngen exec NGEN64 >> checkResult
let sn32 = exec SN32 >> checkResult
let sn64 = exec SN32 >> checkResult

// rem Disable strong-name validation for F# binaries built from open source that are signed with the microsoft key
// %SN32% -Vr FSharp.Core,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Build,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Compiler.Interactive.Settings,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Compiler.Hosted,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Compiler,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Compiler.Server.Shared,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.Editor,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.LanguageService,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.LanguageService.Base,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.LanguageService.Compiler,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.ProjectSystem.Base,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.ProjectSystem.FSharp,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.ProjectSystem.PropertyPages,b03f5f7f11d50a3a
// %SN32% -Vr FSharp.VS.FSI,b03f5f7f11d50a3a
// %SN32% -Vr Unittests,b03f5f7f11d50a3a
// %SN32% -Vr Salsa,b03f5f7f11d50a3a

let strongName (snExe: string -> Result<_,_>) = processor {
let all =
[ "FSharp.Core";
"FSharp.Build";
"FSharp.Compiler.Interactive.Settings";"FSharp.Compiler.Hosted";
"FSharp.Compiler";"FSharp.Compiler.Server.Shared";
"FSharp.Editor";
"FSharp.LanguageService";"FSharp.LanguageService.Base";"FSharp.LanguageService.Compiler";
"FSharp.ProjectSystem.Base";"FSharp.ProjectSystem.FSharp";"FSharp.ProjectSystem.PropertyPages";
"FSharp.VS.FSI";
"Unittests";
"Salsa" ]
for a in all do
do! snExe (sprintf " -Vr %s,b03f5f7f11d50a3a" a)
}

do! strongName sn32

//if /i "%PROCESSOR_ARCHITECTURE%"=="AMD64" (
do! if processorArchitecture = AMD64 then
// %SN64% -Vr FSharp.Core,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Build,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Compiler.Interactive.Settings,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Compiler.Hosted,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Compiler,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Compiler.Server.Shared,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.Editor,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.LanguageService,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.LanguageService.Base,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.LanguageService.Compiler,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.ProjectSystem.Base,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.ProjectSystem.FSharp,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.ProjectSystem.PropertyPages,b03f5f7f11d50a3a
// %SN64% -Vr FSharp.VS.FSI,b03f5f7f11d50a3a
// %SN64% -Vr Unittests,b03f5f7f11d50a3a
// %SN64% -Vr Salsa,b03f5f7f11d50a3a
strongName sn64
else
(fun () -> Success ())
//)

// rem Only GACing FSharp.Core for now
// %GACUTIL% /if %BINDIR%\FSharp.Core.dll
do! gacutil "/if" (binDir/"FSharp.Core.dll")

// rem NGen fsc, fsi, fsiAnyCpu, and FSharp.Build.dll
// if /i not "%2"=="-ngen" goto :donengen

if args.Ngen then
// "%NGEN32%" install "%BINDIR%\fsc.exe" /queue:1
// "%NGEN32%" install "%BINDIR%\fsi.exe" /queue:1
// "%NGEN32%" install "%BINDIR%\FSharp.Build.dll" /queue:1
// "%NGEN32%" executeQueuedItems 1
do! ngen32 [binDir/"fsc.exe"; binDir/"fsi.exe"; binDir/"FSharp.Build.dll"]

// if /i "%PROCESSOR_ARCHITECTURE%"=="AMD64" (
if processorArchitecture = AMD64 then
// "%NGEN64%" install "%BINDIR%\fsiAnyCpu.exe" /queue:1
// "%NGEN64%" install "%BINDIR%\FSharp.Build.dll" /queue:1
// "%NGEN64%" executeQueuedItems 1
do! ngen64 [binDir/"fsiAnyCpu.exe"; binDir/"FSharp.Build.dll"]
// )
//:donengen

}
54 changes: 49 additions & 5 deletions tests/RunTests.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,35 @@ if not exist "%~dp0%..\packages\NUnit.Runners.2.6.4\tools\" (

rem "ttags" indicates what test areas will be run, based on the tags in the test.lst files
set TTAGS_ARG=
SET TTAGS=
set _tmp=%3
if not '%_tmp%' == '' set TTAGS_ARG=-ttags:%_tmp:"=%
if not '%_tmp%' == '' (
set TTAGS_ARG=-ttags:%_tmp:"=%
set TTAGS=%_tmp:"=%
)

rem "nottags" indicates which test areas/test cases will NOT be run, based on the tags in the test.lst and env.lst files
set NO_TTAGS_ARG=-nottags:ReqPP,NOOPEN
set NO_TTAGS=ReqPP,NOOPEN
set _tmp=%4
if not '%_tmp%' == '' set NO_TTAGS_ARG=-nottags:ReqPP,NOOPEN,%_tmp:"=%
if not '%_tmp%' == '' (
set NO_TTAGS_ARG=-nottags:ReqPP,NOOPEN,%_tmp:"=%
set NO_TTAGS=ReqPP,NOOPEN,%_tmp:"=%
)

if /I "%APPVEYOR_CI%" == "1" (set NO_TTAGS_ARG=%NO_TTAGS_ARG%,NO_CI)
if /I "%APPVEYOR_CI%" == "1" (
set NO_TTAGS_ARG=%NO_TTAGS_ARG%,NO_CI
set NO_TTAGS=%NO_TTAGS%,NO_CI
)

set PARALLEL_ARG=-procs:%NUMBER_OF_PROCESSORS%

rem This can be set to 1 to reduce the number of permutations used and avoid some of the extra-time-consuming tests
set REDUCED_RUNTIME=1
if "%REDUCED_RUNTIME%" == "1" set NO_TTAGS_ARG=%NO_TTAGS_ARG%,Expensive
set REDUCED_RUNTIME=0
if "%REDUCED_RUNTIME%" == "1" (
set NO_TTAGS_ARG=%NO_TTAGS_ARG%,Expensive
set NO_TTAGS=%NO_TTAGS%,Expensive
)

rem Set this to 1 in order to use an external compiler host process
rem This only has an effect when running the FSHARPQA tests, but can
Expand Down Expand Up @@ -89,6 +103,10 @@ exit /b 1

:FSHARP

if not '%FSHARP_TEST_SUITE_USE_NUNIT_RUNNER%' == '' (
goto :FSHARP_NUNIT
)

set RESULTFILE=FSharp_Results.log
set FAILFILE=FSharp_Failures.log
set FAILENV=FSharp_Failures
Expand All @@ -107,6 +125,32 @@ echo perl %~dp0\fsharpqa\testenv\bin\runall.pl -resultsroot %RESULTSDIR% -result
goto :EOF


:FSHARP_NUNIT

set FSHARP_TEST_SUITE_CONFIGURATION=%FLAVOR%

set XMLFILE=%RESULTSDIR%\FSharpNunit_Xml.xml
set OUTPUTFILE=%RESULTSDIR%\FSharpNunit_Output.log
set ERRORFILE=%RESULTSDIR%\FSharpNunit_Error.log

setlocal EnableDelayedExpansion

set TTAGS_NUNIT_ARG=
if not '!TTAGS!' == '' (set TTAGS_NUNIT_ARG=--include="!TTAGS!")

set NO_TTAGS_NUNIT_ARG=
if not '!NO_TTAGS!' == '' (set NO_TTAGS_NUNIT_ARG=--exclude="!NO_TTAGS!")

SET NUNIT3_CONSOLE=%~dp0%..\packages\NUnit.Console.3.0.0-beta-3\tools\nunit-console.exe

echo "%NUNIT3_CONSOLE%" "%FSCBINPATH%\..\..\net40\bin\FSharp.Tests.FSharp.dll" --framework:V4.0 !TTAGS_NUNIT_ARG! !NO_TTAGS_NUNIT_ARG! --work="%RESULTSDIR%" --output="%OUTPUTFILE%" --err="%ERRORFILE%" --result="%XMLFILE%"

"%NUNIT3_CONSOLE%" "%FSCBINPATH%\..\..\net40\bin\FSharp.Tests.FSharp.dll" --framework:V4.0 !TTAGS_NUNIT_ARG! !NO_TTAGS_NUNIT_ARG! --work="%RESULTSDIR%" --output="%OUTPUTFILE%" --err="%ERRORFILE%" --result="%XMLFILE%"


goto :EOF


:FSHARPQA

set OSARCH=%PROCESSOR_ARCHITECTURE%
Expand Down
Loading