MueLu: Cuda build errors with complex enabled #5151

vbrunini · 2019-05-10T15:04:22Z

Bug Report

@trilinos/MueLu

Description

FAILED: packages/muelu/src/CMakeFiles/muelu.dir/Utils/ExplicitInstantiation/MueLu_Utilities_kokkos.cpp.o
/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: calling a constexpr __host__ function("std::complex<double> ::complex") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: identifier "std::complex<double> ::complex" is undefined in device code

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: calling a constexpr __host__ function("std::complex<double> ::__rep const") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: identifier "std::complex<double> ::__rep const" is undefined in device code

/sierra/sntools/SDK/compilers/gcc/7.2.0-RHEL6/include/c++/7.2.0/complex(597): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: calling a constexpr __host__ function("std::complex<double> ::complex") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: identifier "std::complex<double> ::complex" is undefined in device code

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: calling a constexpr __host__ function("std::complex<double> ::__rep const") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/scratch/vebruni/src/code_clean/TPLs_src/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(474): error: identifier "std::complex<double> ::__rep const" is undefined in device code

/sierra/sntools/SDK/compilers/gcc/7.2.0-RHEL6/include/c++/7.2.0/complex(597): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::,  ::Kokkos::RangePolicy<int,  ::Kokkos::Cuda > ,  ::Kokkos::Cuda> ::operator () const") is not allowed

10 errors detected in the compilation of "/tmp/tmpxft_000085a8_00000000-6_MueLu_Utilities_kokkos.cpp1.ii".

Steps to Reproduce

SHA1: cbb3aff
Configure script:

cmake -G "Ninja" \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
  -D Trilinos_EXTRA_LINK_FLAGS:STRING="" \
  -D CMAKE_BUILD_TYPE:STRING=release \
  -D LAPACK_LIBRARY_DIRS:PATH=/path/to/lapack
  -D LAPACK_LIBRARY_NAMES:STRING="src-lapack-c;src-lapack-d;src-lapack-s;src-lapack-z" \
  -D BLAS_LIBRARY_DIRS:PATH=/path/to/blas
  -D BLAS_LIBRARY_NAMES:STRING="src-blas-c;src-blas-d;src-blas-s;src-blas-z" \
  -D Trilinos_ENABLE_OpenMP:BOOL=OFF \
  -D Kokkos_ENABLE_Cuda_Relocatable_Device_Code:BOOL=ON \
  -D TPL_ENABLE_CUDA:BOOL=ON \
  -D KOKKOS_ARCH:STRING=Pascal60 \
  -D KOKKOS_ENABLE_CUDA_UVM:BOOL=ON \
  -D KOKKOS_ENABLE_CUDA_LAMBDA:BOOL=ON \
  -DCMAKE_C_COMPILER=/projects/sierra/linux_rh6/SDK/mpi/openmpi/1.10.2-nvidia-9.2.88-RHEL6/bin/mpicc \
  -DCMAKE_CXX_COMPILER=/projects/sierra/linux_rh6/SDK/mpi/openmpi/1.10.2-nvidia-9.2.88-RHEL6/bin/mpicxx \
  -DCMAKE_Fortran_COMPILER=/projects/sierra/linux_rh6/SDK/mpi/openmpi/1.10.2-nvidia-9.2.88-RHEL6/bin/mpif90 \
  -DKokkos_ENABLE_SIERRA_BUILD:BOOL=OFF \
  -DTrilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF \
  -DCMAKE_INSTALL_PREFIX:PATH="/path/to/install" \
  -DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
  -DTrilinos_ENABLE_Gtest:BOOL=OFF \
  -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
  -DTPL_ENABLE_MPI:BOOL=ON \
  -DCMAKE_EXTRA_LINK_FLAGS:STRING= \
  -DCMAKE_C_FLAGS_RELEASE_OVERRIDE:STRING="-DNDEBUG" \
  -DCMAKE_CXX_FLAGS_RELEASE_OVERRIDE:STRING="-DNDEBUG" \
  -DCMAKE_Fortran_FLAGS_RELEASE_OVERRIDE:STRING="-DNDEBUG" \
  -DCMAKE_C_FLAGS:STRING="-O2 -finline-functions -ftemplate-depth-128 -Wno-inline -w " \
  -DCMAKE_CXX_FLAGS:STRING="-O2 -finline-functions -ftemplate-depth-128 -Wno-inline -w " \
  -DCMAKE_Fortran_FLAGS:STRING="-O2 -finline-functions -ftemplate-depth-128 -Wno-inline -w  " \
  -DTrilinos_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
  -DTPL_ENABLE_Pthread:BOOL=OFF \
  -DHAVE_GCC_ABI_DEMANGLE:BOOL=ON \
  -DTrilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
  -DTPL_ENABLE_ParMETIS:BOOL=ON \
  -DParMETIS_INCLUDE_DIRS:PATH="/path/to/parmetis" \
  -DParMETIS_LIBRARY_DIRS:PATH="/path/to/parmetis" \
  -DMETIS_INCLUDE_DIRS:PATH="/path/to/metis" \
  -DMETIS_LIBRARY_DIRS:PATH="/path/to/metis" \
  -DTPL_ENABLE_UMFPACK:BOOL=ON \
  -DUMFPACK_INCLUDE_DIRS:PATH="/path/to/umfpack" \
  -DUMFPACK_LIBRARY_DIRS:PATH="/path/to/umfpack" \
  -DTPL_ENABLE_y12m:BOOL=ON \
  -Dy12m_LIBRARY_DIRS:PATH="/path/to/y12m" \
  -DTPL_ENABLE_Boost:BOOL=ON \
  -DTPL_Boost_INCLUDE_DIRS:PATH="/path/to/boost" \
  -DTPL_ENABLE_yaml-cpp:BOOL=OFF \
  -Dyaml-cpp_INCLUDE_DIRS:PATH="/path/to/yaml" \
  -Dyaml-cpp_LIBRARY_DIRS:PATH="/path/to/yaml" \
  -DSuperLU_INCLUDE_DIRS:PATH="/path/to/superlu" \
  -DSuperLU_LIBRARY_DIRS:PATH="/path/to/superlu" \
  -DScotch_INCLUDE_DIRS:PATH="/path/to/scotch" \
  -DScotch_LIBRARY_DIRS:PATH="/path/to/scotch" \
  -DTrilinos_ENABLE_ThreadPool:BOOL=ON \
  -DTrilinos_ENABLE_Zoltan:BOOL=ON \
  -DZoltan_ENABLE_Scotch:BOOL=OFF \
  -DTrilinos_ENABLE_Zoltan2:BOOL=ON \
  -DZoltan2_ENABLE_Scotch:BOOL=OFF \
  -DZoltan2_ENABLE_Experimental:BOOL=ON \
  -DZoltan2_ENABLE_HYPERGRAPHMODEL:BOOL=OFF \
  -DTrilinos_ENABLE_Shards:BOOL=ON \
  -DTrilinos_ENABLE_Mesquite:BOOL=OFF \
  -DTrilinos_ENABLE_Teuchos:BOOL=ON \
  -DTrilinos_ENABLE_RTOp:BOOL=ON \
  -DTrilinos_ENABLE_Epetra:BOOL=ON \
  -DTrilinos_ENABLE_EpetraExt:BOOL=ON \
  -DTrilinos_ENABLE_Triutils:BOOL=ON \
  -DTrilinos_ENABLE_Thyra:BOOL=ON \
  -DTrilinos_ENABLE_AztecOO:BOOL=ON \
  -DTrilinos_ENABLE_Amesos:BOOL=ON \
  -DTrilinos_ENABLE_Amesos2:BOOL=ON \
  -DTrilinos_ENABLE_Belos:BOOL=ON \
  -DTrilinos_ENABLE_Intrepid:BOOL=ON \
  -DTrilinos_ENABLE_IntrepidIntrepid2:BOOL=OFF \
  -DIntrepid_ENABLE_KokkosCore:BOOL=OFF \
  -DTrilinos_ENABLE_Ifpack:BOOL=ON \
  -DTrilinos_ENABLE_Ifpack2:BOOL=ON \
  -DTrilinos_ENABLE_Tpetra:BOOL=ON \
  -DTrilinos_ENABLE_Kokkos:BOOL=ON \
  -DTrilinos_ENABLE_KokkosCore:BOOL=ON \
  -DTrilinos_ENABLE_KokkosContainers:BOOL=ON \
  -DTrilinos_ENABLE_KokkosTSQR:BOOL=OFF \
  -DTrilinos_ENABLE_Anasazi:BOOL=ON \
  -DTrilinos_ENABLE_ML:BOOL=ON \
  -DTrilinos_ENABLE_MueLu:BOOL=ON \
  -DTrilinos_ENABLE_Stratimikos:BOOL=OFF \
  -DTrilinos_ENABLE_NOX:BOOL=ON \
  -DTrilinos_ENABLE_FEI:BOOL=ON \
  -DTrilinos_ENABLE_TrilinosCouplings:BOOL=ON \
  -DTrilinos_ENABLE_MOOCHO:BOOL=OFF \
  -DTrilinos_ENABLE_Sacado:BOOL=ON \
  -DSacado_ENABLE_KokkosCore:BOOL=ON \
  -DTrilinos_ENABLE_Stokhos:BOOL=ON \
  -DTrilinos_ENABLE_Pamgen:BOOL=ON \
  -DPamgen_ENABLE_Boost:BOOL=OFF \
  -DTrilinos_ENABLE_Rythmos:BOOL=ON \
  -DTrilinos_ENABLE_ThyraEpetraExtAdapters:BOOL=ON \
  -DTrilinos_ENABLE_Xpetra:BOOL=ON \
  -DTrilinos_ENABLE_ShyLU_NodeTacho:BOOL=ON \
  -DTacho_ENABLE_INT_INT:BOOL=ON \
  -DAmesos2_ENABLE_ShyLU_NodeTacho:BOOL=OFF \
  -DTrilinos_ENABLE_ShyLU_NodeHTS:BOOL=ON \
  -DTrilinos_ENABLE_ShyLUFastILU:BOOL=ON \
  -DTrilinos_ENABLE_Isorropia:BOOL=OFF \
  -DTrilinos_ENABLE_STKClassic:BOOL=OFF \
  -DTrilinos_ENABLE_STK:BOOL=OFF \
  -DTrilinos_ENABLE_SEACAS:BOOL=OFF \
  -DAmesos_ENABLE_SuperLU:BOOL=ON \
  -DAmesos2_ENABLE_SuperLU:BOOL=ON \
  -DAztecOO_ENABLE_AZLU:BOOL=ON \
  -DTrilinos_ENABLE_COMPLEX_DOUBLE:BOOL=ON \
  -DTrilinos_ENABLE_FLOAT:BOOL=ON \
  -DZoltan_ENABLE_METIS:BOOL=OFF \
  -DStokhos_ENABLE_Ifpack:BOOL=OFF \
  -DStokhos_ENABLE_ML:BOOL=OFF \
  -DStokhos_ENABLE_Anasazi:BOOL=OFF \
  -DStokhos_ENABLE_Sacado:BOOL=ON \
  -DStokhos_ENABLE_NOX:BOOL=OFF \
  -DStokhos_ENABLE_KokkosClassic:BOOL=OFF \
  -DStokhos_ENABLE_KokkosCore:BOOL=OFF \
  -DStokhos_ENABLE_Tpetra:BOOL=OFF \
  -DStokhos_ENABLE_Ifpack2:BOOL=OFF \
  -DStokhos_ENABLE_AztecOO:BOOL=OFF \
  -DStokhos_ENABLE_Stratimikos:BOOL=OFF \
  -DStokhos_ENABLE_Belos:BOOL=OFF \
  -DML_ENABLE_ParMETIS:BOOL=OFF \
  -DML_ENABLE_METIS:BOOL=OFF \
  -DNOX_ENABLE_ABSTRACT_IMPLEMENTATION_THYRA:BOOL=OFF \
  -DKOKKOS_ENABLE_DEPRECATED_CODE:BOOL=OFF \
  -DKOKKOS_ENABLE_PROFILING:BOOL=ON \
  -DTrilinos_ENABLE_LINEAR_SOLVER_FACTORY_REGISTRATION:BOOL=ON \
  -DTrilinos_ENABLE_MiniTensor:BOOL=ON \
  -DTrilinos_ENABLE_Galeri:BOOL=OFF \
  -DTpetra_INST_SERIAL:BOOL=ON \
  -DTpetra_INST_INT_LONG:BOOL=ON \
  -DTpetra_INST_INT_LONG_LONG:BOOL=OFF \
 /path/to/Trilinos

The text was updated successfully, but these errors were encountered:

mhoemmen · 2019-05-10T15:25:19Z

@trilinos/muelu

csiefer2 · 2019-05-13T14:18:00Z

FYI - PRs are blocked right now #5147

csiefer2 · 2019-05-13T21:43:14Z

As per @crtrott , we need to use Kokkos::complex instead of std::complex wherever this is happening. Tpetra has some workaround where it does the translation between the two... we could be grabbing the un-translated guy.

mhoemmen · 2019-05-14T00:16:37Z

Kokkos::ArithTraits<std::complex<T>>::val_type is Kokkos::complex<T>.

vbrunini · 2019-05-21T22:28:04Z

Any update on this?

mhoemmen · 2019-05-21T23:25:32Z

@vbrunini This may have been fixed -- @rrdrake says the version of Trilinos from last week is fine.

rrdrake · 2019-05-22T01:54:03Z

I got a successful build of Trilinos configured for Sierra using the following master SHA1

e4ab0c3295994554ce4bc4ba5015d2012677f904 Merge Pull Request #5169 from trilinos/Trilinos/master_merge_20190514_000635

vbrunini · 2019-05-22T12:53:46Z

I'm still seeing nvidia build errors with develop as of SHA 9fa2e68ac1e4f147d16723b2a14d5a14ee40e27d from yesterday. @rrdrake Did you do an nvidia build with that SHA1?

rrdrake · 2019-05-22T12:59:33Z

@vbrunini Ah, I did not do an NVIDIA build

MueLu: Fix #5151.

…s:develop' (8380623). * trilinos-develop: (78 commits) MueLu: fixing an issue in the structured region driver data Updates to handle when a non RK stepper observer is passed in. (trilinos#5324) Xpetra: add forced map writing to unit test MueLu: updating some tests/unit_tests, see issue trilinos#5310 Xpetra: add option to force writing of all maps Fixes to make stk build when BoostLib is not enabled. Add Intel 18.0.5 compiler to ATDM environment. Tpetra: Fix trilinos#4627 Xpetra: don't write maps unnecessarily Fix trilinos#5151. Fix stk cmake files to disable unit and doc tests that depend on STKNGP_TEST if STKNGP_TEST is not enabled. MueLu: removing the Epetra path tests for unit-tests-kokkos, see issue trilinos#4325 Xpetra: fix MultiVector unit-test for Epetra=OFF configuration, see issue trilinos#5300 fix another type of quoting error TrilinosCouplings: More updates for Avatar-as-external-package Correct shell quoting in the package file replacement for python SEACAS: bug fix for may snapshot Python testing - resolve issues found during testing Stk update (trilinos#5289) Xpetra: fixing a specialization of the Xpetra::TpetraOperator see issue trilinos#5293 ...

…s:develop' (8380623). * trilinos-develop: (81 commits) epetra: Fix for unguarded macro defined in SLU epetra: Fix for unguarded SuperLU macro MueLu: fixing an issue in the structured region driver data Xpetra: ETI conversion for Xpetra_TpetraExport.hpp Updates to handle when a non RK stepper observer is passed in. (trilinos#5324) Xpetra: add forced map writing to unit test MueLu: updating some tests/unit_tests, see issue trilinos#5310 Xpetra: add option to force writing of all maps Fixes to make stk build when BoostLib is not enabled. Add Intel 18.0.5 compiler to ATDM environment. Tpetra: Fix trilinos#4627 Xpetra: don't write maps unnecessarily Fix trilinos#5151. Fix stk cmake files to disable unit and doc tests that depend on STKNGP_TEST if STKNGP_TEST is not enabled. MueLu: removing the Epetra path tests for unit-tests-kokkos, see issue trilinos#4325 Xpetra: fix MultiVector unit-test for Epetra=OFF configuration, see issue trilinos#5300 fix another type of quoting error TrilinosCouplings: More updates for Avatar-as-external-package Correct shell quoting in the package file replacement for python SEACAS: bug fix for may snapshot ...

…s:develop' (8380623). * trilinos-develop: (86 commits) Fixed review erros for AMGX rebuilding MueLu: Avatar interface mods TrilinosCouplings: More cleanup to example Fixed rebuild issue for MueLu+AMGX run epetra: Fix for unguarded macro defined in SLU epetra: Fix for unguarded SuperLU macro MueLu: fixing an issue in the structured region driver data Fix std::complex<> interface to GEEV/GGEVX in Teuchos::LAPACK Xpetra: ETI conversion for Xpetra_TpetraExport.hpp Updates to handle when a non RK stepper observer is passed in. (trilinos#5324) Xpetra: add forced map writing to unit test MueLu: updating some tests/unit_tests, see issue trilinos#5310 Xpetra: add option to force writing of all maps Fixes to make stk build when BoostLib is not enabled. Add Intel 18.0.5 compiler to ATDM environment. Tpetra: Fix trilinos#4627 Xpetra: don't write maps unnecessarily Fix trilinos#5151. Fix stk cmake files to disable unit and doc tests that depend on STKNGP_TEST if STKNGP_TEST is not enabled. MueLu: removing the Epetra path tests for unit-tests-kokkos, see issue trilinos#4325 ...

vbrunini added the type: bug The primary issue is a bug in Trilinos code or tests label May 10, 2019

mhoemmen added the pkg: MueLu label May 10, 2019

vbrunini mentioned this issue Jun 4, 2019

MueLu: Fix #5151. #5308

Merged

mhoemmen added a commit that referenced this issue Jun 5, 2019

Merge pull request #5308 from vbrunini/fix_muelu_nvidia_complex_eti

f60861f

MueLu: Fix #5151.

trilinos-autotester closed this as completed in 0875230 Jun 6, 2019

vbrunini mentioned this issue Jun 10, 2020

Muelu: failed to compile with complex. #7507

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MueLu: Cuda build errors with complex enabled #5151

MueLu: Cuda build errors with complex enabled #5151

vbrunini commented May 10, 2019

mhoemmen commented May 10, 2019

csiefer2 commented May 13, 2019

csiefer2 commented May 13, 2019 •

edited

Loading

mhoemmen commented May 14, 2019

vbrunini commented May 21, 2019

mhoemmen commented May 21, 2019

rrdrake commented May 22, 2019

vbrunini commented May 22, 2019

rrdrake commented May 22, 2019

MueLu: Cuda build errors with complex enabled #5151

MueLu: Cuda build errors with complex enabled #5151

Comments

vbrunini commented May 10, 2019

Bug Report

Description

Steps to Reproduce

mhoemmen commented May 10, 2019

csiefer2 commented May 13, 2019

csiefer2 commented May 13, 2019 • edited Loading

mhoemmen commented May 14, 2019

vbrunini commented May 21, 2019

mhoemmen commented May 21, 2019

rrdrake commented May 22, 2019

vbrunini commented May 22, 2019

rrdrake commented May 22, 2019

csiefer2 commented May 13, 2019 •

edited

Loading