Tpetra: compile problem on Cori

[jhux2]

An ExaWind colleague has reported the following compile error on Cori @ NERSC with Trilinos develop, SHA 3a7d195.

../packages/kokkos/containers/src/Kokkos_DualView.hpp(403): error: static assertion failed with "Template parameter to .sync() must exactly match one of the DualView's device types or one of the execution or memory spaces"
      static_assert(
      ^
          detected during:
            instantiation of "int Kokkos::DualView<DataType, Arg1Type, Arg2Type, Arg3Type>::get_device_side<Device>() [with DataType=double **, Arg1Type=Kokkos::LayoutLeft, Arg2Type=Kokkos::Serial::execution_space, Arg3Type=void, Device=Kokkos::HostSpace::execution_space]" at line 538
            instantiation of "bool Kokkos::DualView<DataType, Arg1Type, Arg2Type, Arg3Type>::need_sync<Device>() const [with DataType=double **, Arg1Type=Kokkos::LayoutLeft, Arg2Type=Kokkos::Serial::execution_space, Arg3Type=void, Device=Kokkos::HostSpace::execution_space]" at line 1519 of "../packages/tpetra/core/src/Tpetra_MultiVector_decl.hpp"
            instantiation of "bool Tpetra::MultiVector<Scalar, LocalOrdinal, GlobalOrdinal, Node>::need_sync<TargetDeviceType>() const [with Scalar=double, LocalOrdinal=int, GlobalOrdinal=int, Node=Kokkos_Compat_KokkosSerialWrapperNode, TargetDeviceType=Kokkos::HostSpace::execution_space]" at line 221 of "../packages/tpetra/core/src/Tpetra_withLocalAccess_MultiVector.hpp"
            instantiation of "Tpetra::Details::GetMasterLocalObject<Tpetra::Details::LocalAccess<Tpetra::Vector<SC, LO, GO, NT>, MemorySpace, am>>::master_local_object_type Tpetra::Details::GetMasterLocalObject<Tpetra::Details::LocalAccess<Tpetra::Vector<SC, LO, GO, NT>, MemorySpace, am>>::get(Tpetra::Details::GetMasterLocalObject<Tpetra::Details::LocalAccess<Tpetra::Vector<SC, LO, GO, NT>, MemorySpace, am>>::local_access_type) [with SC=double, LO=int, GO=int,
                      NT=Kokkos_Compat_KokkosSerialWrapperNode, MemorySpace=Kokkos::HostSpace, am=Tpetra::Details::AccessMode::ReadWrite]" at line 136 of "../packages/tpetra/core/src/Tpetra_withLocalAccess.hpp"
            instantiation of "Tpetra::Details::GetMasterLocalObject<LocalAccessType>::master_local_object_type Tpetra::Details::getMasterLocalObject(LocalAccessType) [with LocalAccessType=Tpetra::Details::LocalAccess<Tpetra::Vector<double, int, int, Kokkos_Compat_KokkosSerialWrapperNode>, Kokkos::HostSpace, Tpetra::Details::AccessMode::ReadWrite>]" at line 533 of "../packages/tpetra/core/src/Tpetra_withLocalAccess.hpp"
            instantiation of "void Tpetra::Details::WithLocalAccess<FirstLocalAccessType, Rest...>::withLocalAccess(Tpetra::Details::WithLocalAccess<FirstLocalAccessType, Rest...>::current_user_function_type, FirstLocalAccessType, Rest...) [with FirstLocalAccessType=Tpetra::Details::LocalAccess<Tpetra::Vector<double, int, int, Kokkos_Compat_KokkosSerialWrapperNode>, Kokkos::HostSpace, Tpetra::Details::AccessMode::ReadWrite>, Rest=<>]" at line 617 of
                      "../packages/tpetra/core/src/Tpetra_withLocalAccess.hpp"
            instantiation of "void Tpetra::withLocalAccess(Tpetra::Details::ArgsToFunction<LocalAccessTypes...>::type, LocalAccessTypes...) [with LocalAccessTypes=<Tpetra::Details::LocalAccess<Tpetra::Vector<double, int, int, Kokkos_Compat_KokkosSerialWrapperNode>, Kokkos::HostSpace, Tpetra::Details::AccessMode::ReadWrite>>]" at line 1103 of "../packages/ifpack2/src/Ifpack2_Relaxation_def.hpp"
            instantiation of "void Ifpack2::Relaxation<MatrixType>::compute() [with MatrixType=Tpetra::RowMatrix<double, int, int, Kokkos_Compat_KokkosSerialWrapperNode>]" at line 61 of "packages/ifpack2/src/Ifpack2_Relaxation_Serial.cpp"

@trilinos/tpetra @sayerhs

[mhoemmen]

Mark Hoemmen [12:03 PM]
I sent a suggestion. My thought was that they have OpenMP enabled in Kokkos, but have only Serial enabled in Tpetra. We don't have any builds that test this condition.
If you aren't using OpenMP at all, then turn it off in the build.
You'll make the build faster that way anyway.
If for some reason you really want that -- say you use OpenMP in Nalu but don't want to build Trilinos' solvers with OpenMP -- let me know and I'll fix this.

[mhoemmen]

@jhux2 I need to know the CMake configuration before I can diagnose the issue.

[jhux2]

@mhoemmen I think you're spot on. I'll attach the configure script.

[jhux2]

config.txt

[jhux2]

Here's the configure output. Looks like OpenMP is enabled in both Kokkos and Tpetra.

-- ****************** Kokkos Settings ******************
-- Execution Spaces
--   Device Parallel: None
--     Host Parallel: OpenMP
--       Host Serial: Serial
-- 
-- Architectures:
--     None
-- 
-- Enabled options
--   KOKKOS_ENABLE_SERIAL
--   KOKKOS_ENABLE_OPENMP
--   KOKKOS_ENABLE_PROFILING

[snip]

Processing enabled package: Tpetra (Classic, TSQR, Core)
-- Tpetra: Enabling deprecated code
-- Determine whether Tpetra will assume that MPI is CUDA aware:
--   - Tpetra_ENABLE_CUDA is OFF, so Tpetra will assume that MPI is not CUDA aware.
-- Tpetra execution space availability (ON means available): 
--   - Serial:  ON
--   - Threads: OFF
--   - OpenMP:  ON
--   - Cuda:    OFF
-- Tpetra: Tpetra_INST_INT_LONG_LONG is enabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED is disabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED_LONG is disabled by default.
-- Tpetra: Tpetra_INST_INT_INT is enabled by default.
-- Tpetra: Tpetra_INST_INT_LONG_LONG is disabled because Tpetra_INST_INT_LONG was enabled by user option.
-- Tpetra: Tpetra_INST_INT_LONG is enabled.

[mhoemmen]

@jhux2 The issue could be caused by Kokkos::DefaultHostExecutionSpace (OpenMP, in this case) differing from Tpetra's default execution space (Serial, in this case).

[mhoemmen]

@jhux2 Fix here #5343 please review.

[jhux2]

@mhoemmen Thank you for fixing this so quickly.