Skip to content

Commit

Permalink
Add support for CMAKE_DO_INSTALL (trilinos/Trilinos#2689)
Browse files Browse the repository at this point in the history 
I added tests for passing and failing installs.  I only implemented this for
the all-at-once approach since that is all we need right now for ATDM
Trilinos.  But we could implement it for the package-by-package mode if needed
without much trouble.

Note that as part of this I fixed an oversight where build failures that did
not cause test failures would be ignored in that all-at-once approach.  That
was wrong.  Now if there are any build or install failures, it will assume
that any (read that 'all') of the tested packages may have failed.  See the
long "NOTE" comment about this.  Long-story-short, this will only really
impact CI builds where there are build failures.  If there are just test
failures (the more common case), then only the packages with failing tests are
flagged as failed and will be enabled on the next CI iteration.
  • Loading branch information
@bartlettroscoe
bartlettroscoe committed Jan 12, 2019
1 parent 373c6d2 commit 0c3bf46
Show file tree
Hide file tree
Showing 4 changed files with 203 additions and 20 deletions.
129 changes: 116 additions & 13 deletions test/ctest_driver/TribitsExampleProject/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_GlobalConfigureFail
"MixedLang FAILED to configure"
"WithSubpackages FAILED to configure"
"Final set packages that failed to configure or have the libraries build: 'SimpleCxx[;]MixedLang[;]WithSubpackages'"
"Final set packages that had any failures: 'SimpleCxx[;]MixedLang[;]WithSubpackages'"
"Final set of packages that had any failures: 'SimpleCxx[;]MixedLang[;]WithSubpackages'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -315,7 +315,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_ST_BreakConfigureOptionalPkg
"CONFIGURE_OPTIONS = '.*[;]-DTribitsExProj_ENABLE_MixedLang:BOOL=OFF[;].*[;]-DTribitsExProj_ENABLE_WrapExternal:BOOL=ON'"
"WrapExternal: Configure passed"
"Final set packages that failed to configure or have the libraries build: 'MixedLang'"
"Final set packages that had any failures: 'MixedLang'"
"Final set of packages that had any failures: 'MixedLang'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -360,7 +360,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_ST_BreakBuildLibOptionalPkg
"CONFIGURE_OPTIONS = '.*[;]-DTribitsExProj_ENABLE_MixedLang:BOOL=OFF[;].*[;]-DTribitsExProj_ENABLE_WrapExternal:BOOL=ON'"
"WrapExternal: Configure passed"
"Final set packages that failed to configure or have the libraries build: 'MixedLang'"
"Final set packages that had any failures: 'MixedLang'"
"Final set of packages that had any failures: 'MixedLang'"
"CMake Error at .*/TribitsCTestDriverCore.cmake"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
Expand Down Expand Up @@ -405,7 +405,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_ST_BreakBuildAllOptionalPkg
"WithSubpackagesA_test_of_a [.]+ +Passed"
"WithSubpackagesC_test_of_c_b_mixed_lang [.]+ +Passed"
"WrapExternal: Configure passed"
"Final set packages that had any failures: 'MixedLang'"
"Final set of packages that had any failures: 'MixedLang'"
"CMake Error at .*/TribitsCTestDriverCore.cmake"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
FAIL_REGULAR_EXPRESSION
Expand Down Expand Up @@ -454,7 +454,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_ST_BreakTestPkg
"WithSubpackagesC_test_of_c_b_mixed_lang [.]+ +Passed"
"WrapExternal: All build passed"
"WrapExternal_run_external_func [.]+ +Passed"
"Final set packages that had any failures: 'SimpleCxx'"
"Final set of packages that had any failures: 'SimpleCxx'"
"CMake Error at .*/TribitsCTestDriverCore.cmake"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
FAIL_REGULAR_EXPRESSION
Expand Down Expand Up @@ -503,7 +503,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_ST_BreakConfigureRequiredPkg
"WrapExternal: Configure passed"
"WrapExternal: All build passed"
"Final set packages that failed to configure or have the libraries build: 'SimpleCxx'"
"Final set packages that had any failures: 'SimpleCxx'"
"Final set of packages that had any failures: 'SimpleCxx'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -733,6 +733,7 @@ TRIBITS_ADD_ADVANCED_TEST( CTestDriver_PBP_PT_ALL_PASS_CALLS_2
"-- CTEST_DO_UPDATES='OFF'"
"-- CTEST_DO_CONFIGURE='TRUE'"
"-- CTEST_DO_BUILD='TRUE'"
"-- CTEST_DO_INSTALL='FALSE'"
"-- ENV_CTEST_DO_TEST='OFF'"
"-- CTEST_DO_TEST='OFF'"
"-- CTEST_DO_COVERAGE_TESTING='FALSE'"
Expand Down Expand Up @@ -1161,7 +1162,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"Skipping build because configure failed"
"Skipping tests because configure failed"
"Done with the all-at-once configure, build, test, ans submit of TribitsExProj packages"
"Final set packages that had any failures: 'ALL_PACKAGES'"
"Final set of packages that had any failures: 'ALL_PACKAGES'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
TEST_3
Expand Down Expand Up @@ -1204,7 +1205,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"Skipping build because configure failed"
"Skipping tests because configure failed"
"Done with the all-at-once configure, build, test, ans submit of TribitsExProj packages"
"Final set packages that had any failures: 'SimpleCxx[;]WrapExternal'"
"Final set of packages that had any failures: 'SimpleCxx[;]WrapExternal'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -1251,7 +1252,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"WithSubpackagesC_test_of_c_b_mixed_lang [.]+[*][*][*]Not Run"
"WrapExternal_run_external_func [.]+[*][*][*]Not Run"
"File '.+/${PACKAGE_NAME}_CTestDriver_AAO${AAO_POSTFIX}_ST_ALL_BreakWithSubpackagesALib/BUILD/Testing/Temporary/LastTestsFailed_.+.log' exists so there were non-passing tests"
"Final set packages that had any failures: 'WithSubpackages.*WrapExternal"
"Final set of packages that had any failures: 'ALL_PACKAGES'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
TEST_3
Expand All @@ -1260,7 +1261,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
ARGS
BUILD/failedPackages.txt
PASS_REGULAR_EXPRESSION_ALL
"WithSubpackages[;]WrapExternal"
"ALL_PACKAGES"
ALWAYS_FAIL_ON_NONZERO_RETURN
)
# NOTE: The above test ensures that the library build failure of an
Expand Down Expand Up @@ -1305,7 +1306,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"SimpleCxx_HelloWorldProg [.]+ +Passed"
"WrapExternal_run_external_func [.]+[*][*][*]Not Run"
"File '.+/${PACKAGE_NAME}_CTestDriver_AAO${AAO_POSTFIX}_ST_PackagesSubset_BreakWithSubpackagesALib/BUILD/Testing/Temporary/LastTestsFailed_.+.log' exists so there were non-passing tests"
"Final set packages that had any failures: 'WrapExternal'"
"Final set of packages that had any failures: 'SimpleCxx[;]WrapExternal'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -1348,7 +1349,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"WithSubpackagesC_test_of_c_util [.]+ +Passed"
"WithSubpackagesC_test_of_c_b_mixed_lang [.]+ +Passed"
"WrapExternal_run_external_func [.]+ +Passed"
"Final set packages that had any failures: 'SimpleCxx'"
"Final set of packages that had any failures: 'ALL_PACKAGES'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -1390,7 +1391,7 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
"WithSubpackagesC_test_of_c_util [.]+ +Passed"
"WithSubpackagesC_test_of_c_b_mixed_lang [.]+ +Passed"
"WrapExternal_run_external_func [.]+ +Passed"
"Final set packages that had any failures: 'SimpleCxx'"
"Final set of packages that had any failures: 'SimpleCxx'"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)
Expand Down Expand Up @@ -1820,6 +1821,108 @@ FUNCTION(GENERATE_AAO_TESTS USE_NEW_AAO_CTEST_CDASH_FEATURES)
# NOTE: The above test does the build in the first ctest -S invocation and
# the tests in teh second ctest -S invocation.


#
# Test CTEST_DO_INSTALL
#

SET(TEST_NAME_BODY CTestDriver_AAO${AAO_POSTFIX}_CTEST_DO_INSTALL_PASS)
SET(TEST_NAME ${PACKAGE_NAME}_${TEST_NAME_BODY})
TRIBITS_ADD_ADVANCED_TEST( ${TEST_NAME_BODY}
OVERALL_WORKING_DIRECTORY TEST_NAME
OVERALL_NUM_MPI_PROCS 1
TEST_0
MESSAGE "Symlink TribitsExampleProject so no need to clone (which would not work)."
CMND ln
ARGS -s ${TribitsExProj_DIR} .
TEST_1
MESSAGE "Run ctest driver testing everyting but no submit to CDash."
CMND env
ARGS
CTEST_DASHBOARD_ROOT=PWD
${AAO_COMMON_ENV_ARGS}
TribitsExProj_ENABLE_SECONDARY_TESTED_CODE=TRUE
TribitsExProj_PACKAGES=SimpleCxx,WithSubpackages
TribitsExProj_CMAKE_INSTALL_PREFIX=install
CTEST_DO_INSTALL=ON
CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE
${CTEST_DROP_SITE_ENV_ARGS}
CTEST_BUILD_NAME=${TEST_NAME}
${CTEST_S_SCRIPT_ARGS}
PASS_REGULAR_EXPRESSION_ALL
"ENV_CTEST_DO_INSTALL='ON'"
"CTEST_DO_INSTALL='ON'"
"Final set of enabled packages: SimpleCxx MixedLang WithSubpackages 3"
"Installing [(]i.e. building target 'install'[)] [.][.][.]"
"Build install output: BUILD_INSTALL_NUM_ERRORS='0',BUILD_INSTALL_RETURN_VAL='0'"
"Install PASSED!"
"File '' does NOT exist so all tests passed!"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL PASSSED"
ALWAYS_FAIL_ON_NONZERO_RETURN
TEST_2
MESSAGE "Run the installed simplecxx-helloworld program."
CMND ./BUILD/install/bin/simplecxx-helloworld
PASS_REGULAR_EXPRESSION_ALL
"Hello World"
"Debug is enabled!"
"Sqr[(]3[)] = 9"
ALWAYS_FAIL_ON_NONZERO_RETURN
TEST_3
MESSAGE "look for some installed header files"
CMND ls ARGS BUILD/install/include
PASS_REGULAR_EXPRESSION_ALL
"A.hpp"
"B.hpp"
"Makefile.export.MixedLang"
"Makefile.export.SimpleCxx"
"Makefile.export.TribitsExProj"
"Makefile.export.WithSubpackages"
"MixedLang.hpp"
"SimpleCxx_HelloWorld.hpp"
ALWAYS_FAIL_ON_NONZERO_RETURN
)

SET(TEST_NAME_BODY CTestDriver_AAO${AAO_POSTFIX}_CTEST_DO_INSTALL_FAIL)
SET(TEST_NAME ${PACKAGE_NAME}_${TEST_NAME_BODY})
SET(TEST_DIR "${CMAKE_CURRENT_BINARY_DIR}/${TEST_NAME}")
TRIBITS_ADD_ADVANCED_TEST( ${TEST_NAME_BODY}
OVERALL_WORKING_DIRECTORY TEST_NAME
OVERALL_NUM_MPI_PROCS 1
TEST_0
MESSAGE "Copy TribitsExampleProject in so we can break it the install."
CMND cp
ARGS -r ${TribitsExProj_DIR} .
TEST_1
MESSAGE "Add the install of a file that does not exist!"
CMND ${CMAKE_CURRENT_SOURCE_DIR}/append_file_with_line.sh
ARGS
${TEST_DIR}/TribitsExampleProject/packages/simple_cxx/src/CMakeLists.txt
"\nINSTALL(FILES does_not_exist.txt DESTINATION dummy_intall_dir)"
TEST_2
MESSAGE "Run ctest driver which includes the install which should fail."
CMND env
ARGS
CTEST_DASHBOARD_ROOT=PWD
${AAO_COMMON_ENV_ARGS}
TribitsExProj_ENABLE_SECONDARY_TESTED_CODE=TRUE
TribitsExProj_PACKAGES=SimpleCxx,WithSubpackages
TribitsExProj_CMAKE_INSTALL_PREFIX=install
CTEST_DO_INSTALL=ON
CTEST_EXPLICITLY_ENABLE_IMPLICITLY_ENABLED_PACKAGES=FALSE
${CTEST_DROP_SITE_ENV_ARGS}
CTEST_BUILD_NAME=${TEST_NAME}
${CTEST_S_SCRIPT_ARGS}
PASS_REGULAR_EXPRESSION_ALL
"ENV_CTEST_DO_INSTALL='ON'"
"CTEST_DO_INSTALL='ON'"
"Final set of enabled packages: SimpleCxx MixedLang WithSubpackages 3"
"Installing [(]i.e. building target 'install'[)] [.][.][.]"
"Build install output: BUILD_INSTALL_NUM_ERRORS='1',BUILD_INSTALL_RETURN_VAL='2'"
"Install FAILED!"
"File '' does NOT exist so all tests passed!"
"TRIBITS_CTEST_DRIVER: OVERALL: ALL FAILED"
ALWAYS_FAIL_ON_ZERO_RETURN
)

ENDFUNCTION()

Expand Down
34 changes: 28 additions & 6 deletions tribits/ctest_driver/TribitsCTestDriverCore.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -281,15 +281,18 @@ INCLUDE(${CMAKE_CURRENT_LIST_DIR}/TribitsCTestDriverCoreHelpers.cmake)
# 8) **Build configured packages and the enabled package tests**. Submit
# "Build" data to CDash (only if `CTEST_DO_BUILD`_ ``= TRUE``).
#
# 9) **Run enabled tests for the configured packages** (only if
# `CTEST_DO_TEST`_ = ``TRUE``). (Also, this will generate coverage data if
# `CTEST_DO_COVERAGE_TESTING`_ ``= TRUE``). Submit "Test" data to CDash.
# 9) **Install the configured and build targets**. Submit
# "Build" install data to CDash (only if `CTEST_DO_INSTALL`_ ``= TRUE``).
#
# 10) **Collect coverage results from tests already run** (only if
# 10) **Run enabled tests for the configured packages** (only if
# `CTEST_DO_TEST`_ = ``TRUE``). (Also, this will generate coverage data if
# `CTEST_DO_COVERAGE_TESTING`_ ``= TRUE``). Submit "Test" data to CDash.
#
# 11) **Collect coverage results from tests already run** (only if
# `CTEST_DO_COVERAGE_TESTING`_ ``= TRUE``). Submit "Coverage" data to
# CDash.
#
# 11) **Run dynamic analysis testing on defined test suite** (e.g. run
# 12) **Run dynamic analysis testing on defined test suite** (e.g. run
# ``valgrind`` with each of the test commands (only if
# `CTEST_DO_MEMORY_TESTING`_ ``= TRUE``). Submit "MemCheck" data to CDash.
#
Expand Down Expand Up @@ -367,6 +370,7 @@ INCLUDE(${CMAKE_CURRENT_LIST_DIR}/TribitsCTestDriverCoreHelpers.cmake)
# * ``CTEST_DO_BUILD`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_CONFIGURE`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_COVERAGE_TESTING`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_INSTALL`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_MEMORY_TESTING`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_NEW_START`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
# * ``CTEST_DO_SUBMIT`` (`Determining what testing-related actions are performed (TRIBITS_CTEST_DRIVER())`_)
Expand Down Expand Up @@ -835,6 +839,20 @@ INCLUDE(${CMAKE_CURRENT_LIST_DIR}/TribitsCTestDriverCoreHelpers.cmake)
# to build) and the option ``-k 999999`` can be used to build all targets
# when there are build failures.
#
# .. _CTEST_DO_INSTALL:
#
# ``CTEST_DO_INSTALL``
#
# If ``TRUE``, then the 'install' target will be built to install what has
# been configured and built by the build step for the all-at-once mode
# (i.e. ``${PROJECT_NAME}_CTEST_DO_ALL_AT_ONCE=TRUE``. If ``FALSE``, no
# install is performed. (NOTE: The cmake var ``CMAKE_INSTALL_PREFIX``
# must be set on the inner cmake configure for this to work correctly.
# Also, the install is currently not implemented for the
# package-by-package mode ``${PROJECT_NAME}_CTEST_DO_ALL_AT_ONCE=FALSE``
# and this option will simply be ignored in that case.) Default
# ``FALSE``.
#
# .. _CTEST_DO_TEST:
#
# ``CTEST_DO_TEST``
Expand Down Expand Up @@ -1470,6 +1488,9 @@ FUNCTION(TRIBITS_CTEST_DRIVER)
# Call CTEST_BUILD(...) or not
SET_DEFAULT_AND_FROM_ENV( CTEST_DO_BUILD TRUE )

# Call CTEST_BUILD( ... install ... ) or not
SET_DEFAULT_AND_FROM_ENV( CTEST_DO_INSTALL FALSE )

# Do the tests or not (Note: must be true for coverage testing)
SET_DEFAULT_AND_FROM_ENV( CTEST_DO_TEST TRUE )

Expand Down Expand Up @@ -2071,7 +2092,8 @@ FUNCTION(TRIBITS_CTEST_DRIVER)

IF(${PROJECT_NAME}_FAILED_PACKAGES)
MESSAGE(
"\nFinal set packages that had any failures: '${${PROJECT_NAME}_FAILED_PACKAGES}'")
"\nFinal set of packages that had any failures:"
" '${${PROJECT_NAME}_FAILED_PACKAGES}'")
ENDIF()

# Write a file listing the packages that failed. This will be read in on the next CI
Expand Down
Loading

0 comments on commit 0c3bf46

Please sign in to comment.