Compilation errors with the GNU environment

[LiuXiaohui123321]

When I try to compile ABACUS(Github version: 050a2c4) with GNU environment,

• cmake-3.16.2
• gcc-7.3.1
• hpcx-2.11.0
• lapack-3.8.0
• scalapack-2.1.0
• fftw-3.3.8
• elpa-2011.11.002

it interrupts with the error that the liblapack.so file was not found.

c++: error: $PathtoLapack/lib/liblapack.so;/$PathtoBlas/lib/libblas.so: No such file or directory

According to the error message, I found the CMakeFiles/abacus.dir/link.txt file,

c++ ... "$PathtoLapack/lib/liblapack.so;/$PathtoBlas/lib/libblas.so" /$PathtoBlas/lib/lib/libblas.so ...

[WHUweiqingzhou]

@denghuilu, could you have a look?

[denghuilu]

Are you in Sugon environment ? It would be better to tell us the cmake commands as well as the loaded modules. Also I would suggest to compile with a self build OpenBLAS library.

[WHUweiqingzhou]

@caic99, could you have a look?

[caic99]

@caic99, could you have a look?

@LiuXiaohui123321 Would you attach your full input script and error log? Thanks.

[LiuXiaohui123321]

Are you in Sugon environment ? It would be better to tell us the cmake commands as well as the loaded modules. Also I would suggest to compile with a self build OpenBLAS library.

module purge
module load compiler/devtoolset/7.3.1 mpi/hpcx/2.11.0/gcc-7.3.1 compiler/cmake/3.16.2
cmake -B build-cpu-gcc7.3.1-hpcx2.11 -DCereal_INCLUDE_DIR=$PathtoCereal/cereal-1.3.0/include -DELPA_LINK_LIBRARIES=$PathtoElpa/lib/libelpa.so -DELPA_INCLUDE_DIRS=$PathtoElpa/include/elpa-2021.11.002 -DFFTW3_OMP_LIBRARY=$PathtoFftw/lib/libfftw3_omp.a -DFFTW3_LIBRARY=$PathtoFftw/lib/libfftw3.a -DFFTW3_FLOAT_LIBRARY=$PathtoFftw/lib/libfftw3f.a -DFFTW3_INCLUDE_DIR=$PathtoFftw/include -DBLAS_DIR=$PathtoBlas/lib -DBLAS_LIBRARY=$PathtoBlas/lib/libblas.so -DLAPACK_DIR=$PathtoLapack/lib -DLAPACK_LIBRARY=$PathtoLapack/lib/liblapack.so -DScaLAPACK_LIBRARY=$PathtoScalapack/lib/libscalapack.so
cmake --build build-cpu-gcc7.3.1-hpcx2.11/ -j4

[LiuXiaohui123321]

@caic99, could you have a look?

@LiuXiaohui123321 Would you attach your full input script and error log? Thanks.

cmake-script-log-error.tar.gz

The attached files are,

• cmake.sh: compilation environment and cmake commands
• cmake-B.log: log of command cmake -B ...
• cmake--build.log: log of command cmake --build ...
• CMakeFiles/ : output directory of cmake commands

[LiuXiaohui123321]

If I change some part of the CMakeFiles/abacus.dir/link.txt file,
from

"$PathtoLapack/lib/liblapack.so;$PathtoBlas/lib/libblas.so"

to

$PathtoLapack/lib/liblapack.so

and the compilation is ok.
So I think there's something wrong with the way cmake provides links to the lapack libraries

[LiuXiaohui123321]

Are you in Sugon environment ? It would be better to tell us the cmake commands as well as the loaded modules. Also I would suggest to compile with a self build OpenBLAS library.

module purge
module load compiler/devtoolset/7.3.1 mpi/hpcx/2.11.0/gcc-7.3.1 compiler/cmake/3.16.2
cmake -B build-cpu-gcc7.3.1-hpcx2.11 -DCereal_INCLUDE_DIR=$PathtoCereal/cereal-1.3.0/include -DELPA_LINK_LIBRARIES=$PathtoElpa/lib/libelpa.so -DELPA_INCLUDE_DIRS=$PathtoElpa/include/elpa-2021.11.002 -DFFTW3_OMP_LIBRARY=$PathtoFftw/lib/libfftw3_omp.a -DFFTW3_LIBRARY=$PathtoFftw/lib/libfftw3.a -DFFTW3_FLOAT_LIBRARY=$PathtoFftw/lib/libfftw3f.a -DFFTW3_INCLUDE_DIR=$PathtoFftw/include -DBLAS_DIR=$PathtoBlas/lib -DBLAS_LIBRARY=$PathtoBlas/lib/libblas.so -DLAPACK_DIR=$PathtoLapack/lib -DLAPACK_LIBRARY=$PathtoLapack/lib/liblapack.so -DScaLAPACK_LIBRARY=$PathtoScalapack/lib/libscalapack.so
cmake --build build-cpu-gcc7.3.1-hpcx2.11/ -j4

@LiuXiaohui123321 Please replace $PathtoXXX to the correct path for each component.

All of $PathtoXXX are self build,
cmake -B build-cpu-gcc7.3.1-hpcx2.11 -DCereal_INCLUDE_DIR=/public/home/liuxh/software/code/cereal-1.3.0/include -DELPA_LINK_LIBRARIES=/public/home/liuxh/software/code/abacus-develop-3.5.4_intel2017-hpcx2.11-gnulib/solve_elpa-kmpc/lib/elpa-gnu/lib/libelpa.so -DELPA_INCLUDE_DIRS=/public/home/liuxh/software/code/abacus-develop-3.5.4_intel2017-hpcx2.11-gnulib/solve_elpa-kmpc/lib/elpa-gnu/include/elpa-2021.11.002 -DFFTW3_OMP_LIBRARY=/public/home/liuxh/software/lib/fftw3.3.8_devtool7.3.1/lib/libfftw3_omp.a -DFFTW3_LIBRARY=/public/home/liuxh/software/lib/fftw3.3.8_devtool7.3.1/lib/libfftw3.a -DFFTW3_FLOAT_LIBRARY=/public/home/liuxh/software/lib/fftw3.3.8_devtool7.3.1/lib/libfftw3f.a -DFFTW3_INCLUDE_DIR=/public/home/liuxh/software/lib/fftw3.3.8_devtool7.3.1/include -DBLAS_DIR=/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib -DBLAS_LIBRARY=/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib/libblas.so -DLAPACK_DIR=/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib -DLAPACK_LIBRARY=/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib/liblapack.so -DScaLAPACK_LIBRARY=/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/scalapack-2.1.0/build/lib/libscalapack.so

[yizeyi18]

This may come from old cmake's FindLAPACK.cmake, ABACUS did not handle it properly and causes problem. May you try adding 2 lines in cmake/FindLapack.cmake and reconfigure to verify the issue?

--- a/abacus-develop/cmake/FindLapack.cmake
+++ b/abacus-develop/cmake/FindLapack.cmake
@@ -6,9 +6,13 @@ if(DEFINED LAPACK_LIBRARY)
   set(LAPACK_LIBRARIES ${LAPACK_LIBRARY})
 endif()

+message("before finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")
+
 find_package(Blas REQUIRED)
 find_package(LAPACK REQUIRED)

+message("after finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")
+
 if(NOT TARGET LAPACK::LAPACK)
     add_library(LAPACK::LAPACK UNKNOWN IMPORTED)
     set_target_properties(LAPACK::LAPACK PROPERTIES

Edit, remove old build dir, rerun cmake, and look at the output. Here I guess cmake-3.16.2's LAPACK_LIBRARIES would contain both BLAS and LAPACK libraries, which makes bug here.

detailed explaination

For ABACUS, LAPACK and BLAS library are linked with seperate cmake target; they are BLAS::BLAS and LAPACK::LAPACK; they come from CMakeLists.txt => cmake/FindLapack.cmake => cmake's FindLAPACK.cmake, usually created and handled by cmake.

Some old versions of cmake do not create these target. To deal with these versions, default routines of ABACUS are implemented in cmake/FindBlas.cmake and cmake/FindLapack.cmake that creates BLAS::BLAS and LAPACK::LAPACK according to BLAS_LIBRARIES and LAPACK_LIBRARIES(set by cmake's FindBLAS.cmake and FindLAPACK.cmake); these default routines are implemented according to #3706.

[LiuXiaohui123321]

This may come from old cmake's FindLAPACK.cmake, ABACUS did not handle it properly and causes problem. May you try adding 2 lines in cmake/FindLapack.cmake and reconfigure to verify the issue?

--- a/abacus-develop/cmake/FindLapack.cmake
+++ b/abacus-develop/cmake/FindLapack.cmake
@@ -6,9 +6,13 @@ if(DEFINED LAPACK_LIBRARY)
   set(LAPACK_LIBRARIES ${LAPACK_LIBRARY})
 endif()

+message("before finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")
+
 find_package(Blas REQUIRED)
 find_package(LAPACK REQUIRED)

+message("after finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")
+
 if(NOT TARGET LAPACK::LAPACK)
     add_library(LAPACK::LAPACK UNKNOWN IMPORTED)
     set_target_properties(LAPACK::LAPACK PROPERTIES

Edit, remove old build dir, rerun cmake, and look at the output. Here I guess cmake-3.16.2's LAPACK_LIBRARIES would contain both BLAS and LAPACK libraries, which makes bug here.

detailed explaination

Hi! I add the 2 lines, and the output is,

...
before finding LAPACK by cmake, $LAPACK_LIBRARIES is
...
after finding LAPACK by cmake, $LAPACK_LIBRARIES is /public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib/liblapack.so;/public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib/libblas.so

[yizeyi18]

Hi @LiuXiaohui123321 ,
Thanks for your report. For a quick fix, you could add the following lines in the same FindLapack.cmake:

diff --git a/abacus-develop/cmake/FindLapack.cmake b/abacus-develop/cmake/FindLapack1.cmake
index 15c3976..ad3dc05 100644
--- a/abacus-develop/cmake/FindLapack.cmake
+++ b/abacus-develop/cmake/FindLapack1.cmake
@@ -6,10 +6,15 @@ if(DEFINED LAPACK_LIBRARY)
   set(LAPACK_LIBRARIES ${LAPACK_LIBRARY})
 endif()

message("before finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")

 find_package(Blas REQUIRED)
 find_package(LAPACK REQUIRED)

message("after finding LAPACK by cmake, $LAPACK_LIBRARIES is ${LAPACK_LIBRARIES}")

 if(NOT TARGET LAPACK::LAPACK)
+    list(GET LAPACK_LIBRARIES 0 LAPACK_LIBRARIES)
     add_library(LAPACK::LAPACK UNKNOWN IMPORTED)
     set_target_properties(LAPACK::LAPACK PROPERTIES
         IMPORTED_LINK_INTERFACE_LANGUAGES "C"

This would only use the first component, /public/home/liuxh/software/code/gnu/devtoolset7.3.1-hpcx2.11.0/lapack-3.8.0/build/lib/liblapack.so, of the ";" seperated list which list is not supported by the compiler.

The return you report means, for netlib LAPACK/BLAS, cmake-3.16.2 really mixes BLAS and LAPACK libraries in one return value; more details for fixing it, like if other versions of cmake would do the same or in a reverted order, if other BLAS-LAPACK seperated libs would face a same issue(e.g blis/libflame, the libs used in AMD optimizing CPU libraries). Detecting this need our works, you could try find the FindLAPACK.cmake your cmake used and upload it here to help us know what's happening here.

@caic99 Below are some supposed solutions, which one would you, or the core developers prefer?

Possible solutions contains:

• raising minium required cmake version to a version returning a LAPACK::LAPACK target, say, 3.18
• add more default lines in FindLapack.cmake, and keep using cmake's FindLAPACK.cmake as the single source to find LAPACK
  • and also change its file name to a OSX/Win compatible case-insensitive style, say, FindAbacusLAPACK.cmake
• include some default routines to find LAPACK/BLAS while cmake version is low:
  • rolling back to ABACUS-made FindLAPACK.cmake
  • including FindLAPACK.cmake from some newer cmake
  • or including some others' works like https://github.com/wavefunction91/linalg-cmake-modules. This and 2 should be distributed with their BSD license.
• totally use something distributed with ABACUS.

[LiuXiaohui123321]

@yizeyi18 Thanks for your reply and suggestion!
The file FindLAPACK.cmake I used is the same with that in the source code: https://github.com/deepmodeling/abacus-develop/blob/854d49ce5420af1588c60c9fc8365f314f7d50a4/cmake/FindLapack.cmake

[yizeyi18]

Hi @LiuXiaohui123321 ,
Did the added line solve the link command issue?

[LiuXiaohui123321]

Hi @LiuXiaohui123321 , Did the added line solve the link command issue?

Yes, it works.