-
Notifications
You must be signed in to change notification settings - Fork 4.3k
How to Test
The unit tests and end-to-end tests provided in the \Tests folder are run as part of the check-in workflow on the build server. You can run all tests locally to verify that your code changes didn't break any test.
If you followed the install instructions and installed Boost on your machine you should see the unit tests in the Visual Studio Test Explorer. You can run and debug the unit tests from there. We recall the required steps in the following.
How to run and debug end-to-end tests is described further below.
We are using the Boost library (http://www.boost.org/) for unit tests. We are probably going to incorporate the Boost library into CNTK code in the future. We are currently using Boost 1.59. Download and install boost version 1.59 (you need msvc-12.0 binaries) from here: http://sourceforge.net/projects/boost/files/boost-binaries/1.59.0/boost_1_59_0-msvc-12.0-64.exe/download
Set the environment variable BOOST_INCLUDE PATH to your boost installation, e.g.:
BOOST_INCLUDE_PATH=c:\local\boost_1_59_0
Set the environment variable BOOST_LIB_PATH to the boost libraries, e.g.:
BOOST_LIB_PATH=c:\local\boost_1_59_0\lib64-msvc-12.0
To integrate Boost into the Visual Studio test framework you can install a runner for Boost tests in VS from here: https://visualstudiogallery.msdn.microsoft.com/5f4ae1bd-b769-410e-8238-fb30beda987f
Restart Visual Studio and goto to the 'Test --> Test Settings' menu. There set the default processor architecture to x64 and uncheck the option 'Keep Test Execution Engine Running'. In the Test Explorer window select Group by Traits (next to the search field). After a rebuild you should see all unit tests in the Test Explorer. You can run and debug using the context menu.
The CNTK end-to-end tests can be run on Linux and Windows using the python script 'TestDriver.py' located in \Tests\EndToEndTests. Alternatively they can be run and debugged from Visual Studio. In the following we detail:
- How to use the TestDriver.py script.
- Prerequisites to use TestDriver.py on Windows.
- How to run and debug end-to-end tests from Visual studio.
Start a shell (CygWin shell under Windows) and change directory to the Tests\EndToEndTests folder of your local CNTK repository (note: c:\src\CNTK in CygWin is /cygdrive/c/src/CNTK/). Start with one of the following commands to learn the usage and options of the TestDriver:
python TestDriver.py -h
python TestDriver.py run -h
python TestDriver.py list -h
To list all available end-to-end tests run
python TestDriver.py list
To run a single test, for example Image/QuickE2E, execute
python TestDriver.py run Image/QuickE2E
You can add for example '-d gpu' to only run the test using a GPU or '-f debug'
to only run the test using the debug build. See python TestDriver.py run -h
for all options.
To run all tests from the nightly builds execute
python TestDriver.py run -t nightly
- Install the latest version of Python 2.7 (NOT 3.5!) from https://www.python.org/downloads/windows/ Accept default installation options.
- Install CygWin from http://cygwin.com/install.html During installation select "Install from Internet" (default selection). IMPORTANT: At Select Packages screen type "yaml" in the search field and expand the Python section. Select "python-yaml: Python Lib YAML bindings" (NOT "python3-yaml"). Then finish installation.
- Make sure you have Microsoft MPI installed (there should be an environment variable named MSMPI_INC)
Note: Some tests require an environment variable named CNTK_EXTERNAL_TESTDATA_SOURCE_DIRECTORY pointing to where the data resides. If the external data is not available those tests will be skipped.
You can generate the Visual Studio debug command arguments using the -n options
on the TestDriver for a specific end-to-end test:
python TestDriver.py run -n Image/QuickE2E
From the output of the above command you simply copy the 'VS debugging command args' to the command arguments of the CNTK project in Visual Studio (Right click on CNTK project -> Properties -> Configuration Properties -> Debugging -> Command Arguments). Start debugging the CNTK project.