TriBITs update breaks Trilinos builds with HYPRE enabled

[tjfulle]

I believe “Automatic snapshot commit from tribits at bf7c58f6” breaks Trilinos builds when TPL_ENABLE_HYPRE=ON. As evidence, on/around January 23 Alegra’s experimental builds (built against trilinos@develop with TPL_ENABLE_HYPRE=ON) began failing in the cmake step. The actual error is:

Finished configuring Trilinos!
-- If publishing results using Trilinos, please cite us: https://trilinos.github.io/cite.html
-- Configuring done
CMake Error at cmake/tribits/core/package_arch/TribitsAddLibrary.cmake:415 (add_library):
Target "ifpack2" links to target "HYPRE::all_libs" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
packages/ifpack2/src/CMakeLists.txt:413 (TRIBITS_ADD_LIBRARY)

No changes to Ifpack2 in the timeframe touch any HYPRE related code.

HYPRE is not a CMake package and is found via FindTPLHYPRE.cmake. Does something need to be modified in that module to make HYPRE into a first-class CMake target with an associated all_libs?

Below is a selection from the output of a successful install from the beginning of January.

Processing enabled TPL: HYPRE (enabled explicitly, disable with -DTPL_ENABLE_HYPRE=OFF)
-- HYPRE_FINDMOD='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPRE.cmake'
-- File Trace: TPL        INCLUDE    /Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPRE.cmake
TRIBITS_TPL_FIND_INCLUDE_DIRS_AND_LIBRARIES: HYPRE
-- PARSE_REQUIRED_HEADERS='HYPRE.h;HYPRE_config.h'
-- PARSE_REQUIRED_LIBS_NAMES='HYPRE'
-- TPL_HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- TPL_HYPRE_LIBRARIES='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/lib/libHYPRE.a'
-- HYPRE_LIBRARY_NAMES='HYPRE'
-- HYPRE_LIBRARY_DIRS=''
-- HYPRE_LIBRARY_NAMES='HYPRE'
-- REQUIRED_LIBS_NAMES='HYPRE'
-- HYPRE_LIBRARY_DIRS=''
-- HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- PARSE_REQUIRED_HEADERS='HYPRE.h;HYPRE_config.h'
-- TPL_HYPRE_LIBRARIES='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/lib/libHYPRE.a'
-- TPL_HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- TPL_HYPRE_LIBRARY_DIRS=''
-- TPL_HYPRE_NOT_FOUND='FALSE'

FYI: @csiefer2 @skennon10 @bartlettroscoe

[bartlettroscoe]

@tjfulle, can you please post the STDOUT+STDERR from the CMake configure command along with the generated CMakeCache.txt file? That will give me some more clues.

Then I need to find a way to reproduce this. (If that is with Spack that is okay as long as I can use the developer mode to cd to the Trilinos build directory and run cmake manually.)

[tjfulle]

Attached:

• cmake-out.txt
• CMakeCache.txt

I just generated those files in a build on an ascic machine. I can get you access to the machine and/or I can make any modifications you see fit and run it again.

[tjfulle]

@bartlettroscoe : interestingly, in the output attached above the HYPRE library does not seem to be parsed. Eg, in a working build from a few weeks ago I have the following in the output:

Processing enabled TPL: HYPRE (enabled explicitly, disable with -DTPL_ENABLE_HYPRE=OFF)
-- HYPRE_FINDMOD='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPRE.cmake'
-- File Trace: TPL        INCLUDE    /Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPRE.cmake
TRIBITS_TPL_FIND_INCLUDE_DIRS_AND_LIBRARIES: HYPRE
-- PARSE_REQUIRED_HEADERS='HYPRE.h;HYPRE_config.h'
-- PARSE_REQUIRED_LIBS_NAMES='HYPRE'
-- TPL_HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- TPL_HYPRE_LIBRARIES='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/lib/libHYPRE.a'
-- HYPRE_LIBRARY_NAMES='HYPRE'
-- HYPRE_LIBRARY_DIRS=''
-- HYPRE_LIBRARY_NAMES='HYPRE'
-- REQUIRED_LIBS_NAMES='HYPRE'
-- HYPRE_LIBRARY_DIRS=''
-- HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- PARSE_REQUIRED_HEADERS='HYPRE.h;HYPRE_config.h'
-- TPL_HYPRE_LIBRARIES='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/lib/libHYPRE.a'
-- TPL_HYPRE_INCLUDE_DIRS='/Users/tjfulle/Workspace/Projects/Alegra/Development/Sandboxes/og-master/opt/darwin-monterey-m1/apple-clang-13.1.6/hypre-2.18.2-aynvrixccqsma3odexidbfz2a4iqmvbw/include'
-- TPL_HYPRE_LIBRARY_DIRS=''
-- TPL_HYPRE_NOT_FOUND='FALSE'

But in the more recent output I only see

-- HYPRE_TESTGROUP='EX'
-- HYPRE_FINDMOD='/scratch/alegra/work/trilinos@develop/sb/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPRE.cmake'
-- HYPRE_DEPENDENCIES_FILE='/scratch/alegra/work/trilinos@develop/sb/code/Trilinos/cmake/tribits/common_tpls/FindTPLHYPREDependencies.cmake'
-- HYPRE_TPLS_LIST_FILE='/scratch/alegra/work/trilinos@develop/

so, clearly, something in the way tribits processed the HYPRE TPL has changed between January 20 and January 23

[tjfulle]

@bartlettroscoe - I think I have a workaround. In the early January builds I defined -DIfpack2_ENABLE_HYPRE and TriBITs properly inferred that TPL_ENABLE_HYPRE should be set to ON. This does not seem to be the case in the most current develop branch. So, I explicitly set -DTPL_ENABLE_HYPRE=ON and my build is past the cmake phase and appears to be chugging along. I'll make the change in the nightly CI settings and report back tomorrow if the build completes.

[bartlettroscoe]

so, clearly, something in the way tribits processed the HYPRE TPL has changed between January 20 and January 23

@tjfulle, the change should have occurred with the merge of the Trilinos PR #11458 on Jan 17, 2023, 437 PM EST

The configure output is strange because it seems that the HYPRE TPL is not even being processed! My guess is that this is because the configure input is setting -DTrilinos_ENABLE_HYPRE:BOOL=ON is being passed in instead of -DTPL_ENABLE_HYPRE:BOOL=ON. My guess is that the old version of TriBITS was just ignoring Trilinos_ENABLE_HYPRE=ON but the new version of TriBITS takes that into account when doing the TPL dependencies but not when processing the TPLs themselves.

Do you have the configure output from your Trilinos build before Jan 19, 2023? I bet we will see that Trilinos_ENABLE_HYPRE=ON was being completely ignored and support for HYPRE was not being enabled in Trilinos. Therefore, I bet you can just take -DTrilinos_ENABLE_HYPRE:BOOL=ON out of your configure script and things will work just fine. Did Spack create this configure line for Trilinos?

NOTE: The current TriBITS update I am working on in PR TriBITSPub/TriBITS#560 should fully recognize Trilinos_ENABLE_HYPRE=ON as requesting the HYPRE TPL to be enabled and process it correctly.

NOTE TO SELF: I will need to add an entry to tribits/CHANGELOG.md that mentions that setting <Project>_ENABLE_<TPL_NAME>=ON will not enable support for an external package/TPL where before it would not.

[tjfulle]

It is being built by Spack - but with Alegra's own spackage wherein I set Trilinos_ENABLE_HYPRE=ON. I can copy an old config up. In the meantime I just did in the old build output:

grep -i hyper spack-build-out.txt

and I see

-- Setting TPL_ENABLE_HYPRE=ON because Ifpack2_ENABLE_HYPRE=ON

This is absent from the output from the more recent configure. So, yes, Trilinos_ENABLE_HYPRE was wrong and ignored by TriBITs but in the newest snapshot of TriBITs the connection between Ifpack2_ENABLE_HYPRE=ON and TPL_ENABLE_HYPRE=ON is not being made. I updated my CI scripts so Alegra's trilinos build should be ok, but I think this points to a regression in TriBITs

[tjfulle]

I just tried copying the output and it choked my terminal because it was so big...

[bartlettroscoe]

@tjfulle

@bartlettroscoe - I think I have a workaround. In the early January builds I defined -DIfpack2_ENABLE_HYPRE and TriBITs properly inferred that TPL_ENABLE_HYPRE should be set to ON. This does not seem to be the case in the most current develop branch. So, I explicitly set -DTPL_ENABLE_HYPRE=ON and my build is past the cmake phase and appears to be chugging along. I'll make the change in the nightly CI settings and report back tomorrow if the build completes.

Okay, that makes sense. What happened is that the TriBITS code that does the enable/disable of packages and TPLs was upgraded to look at both Trilinos_ENABLE_<tplName> and TPL_ENABLE_<tplName> (and therefore did not set TPL_ENABLE_HYPRE=ON because it saw that Trilinos_ENABLE_HYPRE=ON was already set) but the code that processes the list of enabled TPLs was still looking at just TPL_ENABLE_<tplName> for which TPL_ENABLE_HYPRE was empty (and therefore false).

I will add a unit test for this use case but I believe that the updated code in TriBITSPub/TriBITS#560 (soon to be merged to TriBITS 'master') already will fix this use case.

So, ironically, if the configure had not set -DTrilinos_ENABLE_HYPRE=ON at all (which was completely ignored in old versions of TriBITS and did nothing) then it would have kept working. (It is so hard to anticipate all of these strange use cases that you would never image anyone would do because before recent TriBITS updates, setting Trilinos_ENABLE_<tplName> to any value was completely ignored. And here lies the danger is changing anything in that it may change backward compatibility in surprising ways.)

Thanks for reporting this!

[bartlettroscoe]

I just tried copying the output and it choked my terminal because it was so big...

Indeed, with '-DTrilinos_VERBOSE_CONFIGURE:BOOL=ON' set you can expect a huge amount of output. (When it says VERBOSE it is VERBOSE.)

[bartlettroscoe]

@tjfulle, is this still and issue? NOTE: this use case should be fixed by TriBITSPub/TriBITS#560 once that merges to TriBITS 'master' and snapshotted to Trilinos 'develop'.

[tjfulle]

We modified our spackage file to get around the issue, so we are good.