MKL discovery in latest release

[VictorEijkhout]

My blas specification passes all the cmake tests, but bombs during build.

Is there an easier way to spcify "I have MKL"?

V.
gcc12tril.zip

[VictorEijkhout]

I mean,

[staff trilinos:1608] find $TACC_MKL_DIR -name \*.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-ilp64-seq.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-lp64-seq.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-lp64-seq.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-lp64-iomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-lp64-tbb.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-ilp64-tbb.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-lp64-tbb.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-sdl.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-ilp64-seq.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-ilp64-gomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-lp64-iomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-ilp64-tbb.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-lp64-gomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-ilp64-iomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-ilp64-iomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-dynamic-ilp64-gomp.pc
/opt/intel/oneapi/mkl/2023.1.0/lib/pkgconfig/mkl-static-lp64-gomp.pc

and that's on the PKG_CONFIG_PATH.

[sebrowne]

-- TPL_BLAS_INCLUDE_DIRS=''
-- TPL_BLAS_LIBRARIES=''
-- BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread'
-- BLAS_LIBRARY_DIRS='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64'
-- BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread'
-- REQUIRED_LIBS_NAMES='mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread'
-- Searching for libs in BLAS_LIBRARY_DIRS='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64'
-- Searching for a lib in the set "mkl_intel_lp64":
-- LIBNAME_SET='mkl_intel_lp64'
--   Searching for lib 'mkl_intel_lp64' ...
-- _BLAS_mkl_intel_lp64_LIBRARY='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_lp64.so'
--     Found lib '/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_lp64.so'
-- Searching for a lib in the set "mkl_intel_thread":
-- LIBNAME_SET='mkl_intel_thread'
--   Searching for lib 'mkl_intel_thread' ...
-- _BLAS_mkl_intel_thread_LIBRARY='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_thread.so'
--     Found lib '/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_thread.so'
-- Searching for a lib in the set "mkl_core":
-- LIBNAME_SET='mkl_core'
--   Searching for lib 'mkl_core' ...
-- _BLAS_mkl_core_LIBRARY='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_core.so'
--     Found lib '/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_core.so'
-- Searching for a lib in the set "iomp5":
-- LIBNAME_SET='iomp5'
--   Searching for lib 'iomp5' ...
-- _BLAS_iomp5_LIBRARY='_BLAS_iomp5_LIBRARY-NOTFOUND'
-- NOTE: Did not find a lib in the lib set "iomp5" for the TPL 'BLAS'!
-- TPL_BLAS_LIBRARIES='/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_intel_thread.so;/opt/intel/oneapi/mkl/2023.1.0/lib/intel64/libmkl_core.so'
-- TPL_BLAS_LIBRARY_DIRS=''
-- TPL_BLAS_NOT_FOUND='FALSE'

Looks like it did not find the Intel compiler threading runtime (which explains the link errors about threading symbols). You'll need to add the compiler's 'lib' directory to BLAS_LIBRARY_DIRS and LAPACK_LIBRARY_DIRS, or use the non-threaded version of MKL.

I've had issues with this type of thing repeatedly in the past. Intel does not make MKL very easy to use. For whatever it's worth, what you're doing is more or less what I've done for most CMake-based projects where I wanted to use MKL (including Trilinos).

[sebrowne]

@bartlettroscoe is there a reason that it doesn't fail if it can't find ALL of the specified TPL libraries? I know it will error out if it can't find ANY of them, but it seems like we'd want it to error out unless it found everything that was specified.

[VictorEijkhout]

" what you're doing is more or less what I've done for most CMake-based projects where I wanted to use MKL (including Trilinos)."

Right. But at least in Intel 2023 there is now a ton of .pc files. Have you tried if those solve the problem?

[bartlettroscoe]

@bartlettroscoe is there a reason that it doesn't fail if it can't find ALL of the specified TPL libraries? I know it will error out if it can't find ANY of them, but it seems like we'd want it to error out unless it found everything that was specified.

@sebrowne that is what #10345 is about. See:

• https://tribits.org/doc/TribitsUsersGuide.html#project-name-must-find-all-tpl-libs
• https://docs.trilinos.org/files/TrilinosBuildReference.html#require-all-tpl-libraries-be-found

@VictorEijkhout, I have no experience with *.pc files. If you want, we can update FindTPLBLAS.cmake to allow calling find_package(BLAS) if want to see if the native CMake find module can do a better job. (It is just that in the past, find_package(BLAS) broke some critical customer configurations so we had to back that out. But we can bring back and option to try it if it would help.)

[VictorEijkhout]

Leaving out mkl_thread seems to solve the problem. Kindly comment on whether threaded blas is actually relevant to Trilinos.

[sebrowne]

I'm not really an expert on how the threaded MKL can affect performance. I'm guessing that @csiefer2 may be able to provide better insight? Or at least tag somebody who may?

It does appear that you've got OpenMP enabled in your Trilinos configuration, so in general I think that Intel recommends using the threading library as well, but I know you can use threaded MKL if you're not using threaded code and vice-versa, so I believe it is primarily a performance question.

[jhux2]

Kindly comment on whether threaded blas is actually relevant to Trilinos.

Not necessary. If memory serves, this could actually causes problems if OpenMP is enable in Kokkos. Probably a question best asked on the Kokkos slack, though.

[VictorEijkhout]

Ok.

1. Issue was caused by Intel stowing its libiomp in a different location from the mkl libraries
2. libmkl_thread uses symbols from libiomp
3. With intel compilers you can of course use the -mkl flag and everything is fine
4. For non-intel compiler you need to go hunting around for the various locations
5. But I shouldn't have used libmkl_thread to begin with. Removing that removes the reference to kmpc_barrier and such OMP stuff.

Closing.