Better identify and hanlde the TriBITS primary meta-project packges

[bartlettroscoe]

Currently, the way that I tried to allow a TrBITS meta-project to differentiate its primary meta-project packages from other upstream packages (just there to satisfy dependencies) was to it define the CMake variable ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE for each upstream repo to say that packages within the repo ${REPOSITORY_NAME} should only be enabled if they are explicitly enabled. I used this in CASL VERA for trying to exclude the testing of packages from the repos Trilinos, SCALE, etc.

Enable logic based on just ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE is currently implemented in the help function TRIBITS_REPOSITORY_IMPLICIT_PACKAGE_ENABLE_IS_ALLOWED() and used in various places.

However, the way I implemented this it only really worked correctly when setting ${PROJECT_NAME}_ENABLE_ALL_PACKAGES when using the first iteration of the CI server using TRIBITS_CTEST_DRIVER() implemented in TribitsCTestDriverCore.cmake which calls TRIBITS_APPLY_REPOSITORY_NO_IMPLICIT_PACKAGE_ENABLE_DISABLE(). It also works for Nightly testing as well driven by TRIBITS_CTEST_DRIVER(). However, this does not address the problem very well and does not have any impact on the usage of the checkin-test.py script when correctly turning on software to test when setting up an ACI sync server, for example.

Also, an unfortunate consequence of the way I implemented this what that all implicit package enables for packages in one of these repos would be skipped due to the fact that the function TRIBITS_IMPLICIT_PACKAGE_ENABLE_IS_ALLOWED() also calls TRIBITS_REPOSITORY_IMPLICIT_PACKAGE_ENABLE_IS_ALLOWED(). Now that I think about this, this was a stupid thing to do. It is also why I could not set Trilinos_NO_IMPLICIT_PACKAGE_ENABLE=ON in the inner VERA CMake configuration process because it straight-up broke VERA.

I think what is need to do is to change the name of ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE to ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES and change the name of TRIBITS_REPOSITORY_IMPLICIT_PACKAGE_ENABLE_IS_ALLOWED() to TRIBITS_IS_PRIMARY_META_PROJECT_PACKAGE().

What we really want is a system for identifying and handling the primary meta-project packages that has the following properties:

1. When the meta-project is configured with ${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON, only the SE packages ${PACKAGE_NAME} that return true from TRIBITS_IS_PRIMARY_META_PROJECT_PACKAGE(${PACKAGE_NAME} ...) will be enabled in the first sweep through all of the SE packages. The result of this first sweep will identify the primary meta-project packages. When verbose output is enabled, also print that a package is not being enabled base on this logic.

2. Only SE packages ${PACKAGE_NAME} that return true from TRIBITS_IS_PRIMARY_META_PROJECT_PACKAGE(${PACKAGE_NAME} ...) will be allowed to have their tests/examples implicitly enabled based on ${PROJECT_NAME}_ENABLE_TESTS/EXAMPLES=ON. When verbose output is enabled, also print that a package's tests/examples are being enabled base on this logic.

3. Any packages that are explicitly enabled with ${PROJECT_NAME}_ENABLE_${PACKAGE_NAME}=ON will be enabled irrespective of ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES.

4. Any package tests/examples that are explicitly enabled with ${PACKAGE}_ENABLE_TEST/EXAMPLES=ON will still be enabled.

5. All other implicit package enables due to sweeping upstream and downstream are unaffected by ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES logic. For example, sweeping downstream/forward to find packages that need to be tested based on a change needs to stay intact. Also, sweeping upstream/backward to enable optional packages needs to stay intact (that is not currently true with the current implementation).

The way this would be used to define meta-projects is to set ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES=ON for all upstream repos that are just contributing package libs, etc. However, some packages from a given repo can be consider primary meta-project packages by adding them to the list ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES_EXCEPT=<pkg0>,<pkg1>,.... Any upstream packages that the meta-project is sure that it never wants should be explicitly disabled inside of the meta-project's ProjectName.cmake file (e.g. with ${PROJECT_NAME}_ENABLE_PACKAGES=<pkg0>,<pkg1>,...).

The above behavior would correctly implement the following use cases:

A) Enable all of the tests/examples for the primary meta-project packages to be built and installed complete with tests and examples => Configure with ${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON and ${PROJECT_NAME}_ENABLE_TESTS=ON.

B) Select all of the primary meta-project packages to be explicitly tested in TRIBITS_CTEST_DRIVER() implemented in TribitsCTestDriverCore.cmake => Set ${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON and ${PROJECT_NAME}_ENABLE_TESTS=ON and whatever packages end up with enabled tests get explicitly processed.

C) Enable the tests and examples for all primary meta-project packages that need to be tested for changes in any upstream packages => Configure with the set of changed packages ${PROJECT_NAME}_ENABLE_${CHANGED_PACKAGE}=ON, ${PROJECT_NAME}_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON, and ${PROJECT_NAME}_ENABLE_TESTS=ON. None of the tests will be enabled for the changed packages if TRIBITS_IS_PRIMARY_META_PROJECT_PACKAGE() returns false. This will enable the tests and examples for all primary meta-project packages that are affected by any of the changed upstream packages. This will also perhaps enable some intermediate packages downstream from the changed packages but upstream from the primary meta-project packages but since there are no tests or examples enabled, only a build failure could mess up the testing. Also, such packages would not be explicitly processed in TRIBITS_CTEST_DRIVER(). This would work for the checkin-test.py script since the tests and examples would only be enabled for primary meta-project packages.

Tasks:

1. Create new variable ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES and new function TRIBITS_IS_PRIMARY_META_PROJECT_PACKAGE() and update all of unit the tests that currently test TRIBITS_APPLY_REPOSITORY_NO_IMPLICIT_PACKAGE_ENABLE_DISABLE(). Update these unit tests one at a time until they are all replaced where the logic is built into the basic enable/disable logic.
2. Remove the functions TRIBITS_REPOSITORY_IMPLICIT_PACKAGE_ENABLE_IS_ALLOWED() and TRIBITS_APPLY_REPOSITORY_NO_IMPLICIT_PACKAGE_ENABLE_DISABLE() and the variables ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE and ${REPOSITORY_NAME}_NO_IMPLICIT_PACKAGE_ENABLE_EXCEPT and any unit tests involving TRIBITS_CTEST_DRIVER() that depend on their behavior.
3. Document the change in logic for ${PROJECT_NAME}_ENABLE_ALL_PACKAGES=ON and ${PROJECT_NAME}_ENABLE_TESTS=ON based on ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES and ${REPOSITORY_NAME}_NO_PRIMARY_META_PROJECT_PACKAGES.
4. Fill in the section on multi-repo projects and meta-projects to explain this strategy.

[bartlettroscoe]

@lefebvre, you should consider adding:

SET(Trilinos_NO_PRIMARY_META_PROJECT_PACKAGES TRUE)

to the file:

SCALE/ProjectName.cmake

This will make SCALE_ENABLE_ALL_PACKAGES and SCALE_ENABLE_TESTS work better for SCALE developers.

[lefebvre]

This sounds great. I'll look into it.
On Sep 18, 2014 10:20 AM, "Roscoe A. Bartlett" [email protected]
wrote:

@lefebvre https://github.com/lefebvre, you should consider adding:

SET(Trilinos_NO_PRIMARY_META_PROJECT_PACKAGES TRUE)

to the file:

SCALE/ProjectName.cmake

This will make SCALE_ENABLE_ALL_PACKAGES
http://htmlpreview.github.io/?https://raw.githubusercontent.com/TriBITSPub/TriBITS/master/doc/developers_guide/TribitsDevelopersGuide.html#project-enable-all-packages-enables-all-pt-cond-st-se-packages
and SCALE_ENABLE_TESTS
http://htmlpreview.github.io/?https://raw.githubusercontent.com/TriBITSPub/TriBITS/master/doc/developers_guide/TribitsDevelopersGuide.html#project-enable-tests-only-enables-explicitly-enabled-se-package-tests
work better for SCALE developers.

—
Reply to this email directly or view it on GitHub
#9 (comment).