Guard KOKKOS_ALL_COMPILE_OPTIONS if Cuda is not enabled

[finkandreas]

I'm building trilinos, not Kokkos standalone, and I'm building with a clang compiler, where I enable OpenMP as parallel hostspace, and no device space (i.e. Cuda disabled).

Variable KOKKOS_ALL_COMPILE_OPTIONS is referenced in file cmake/KokkosTrilinosConfig.cmake.in (when building with Trilinos, this triggers a bug)
The installed KokkosConfig.cmake will contain the lines:

  set_target_properties(Kokkos::kokkos PROPERTIES
    INTERFACE_LINK_LIBRARIES "kokkosalgorithms;kokkoscontainers;kokkoscore;-fopenmp"
    INTERFACE_COMPILE_FEATURES "cxx_std_14"
    INTERFACE_COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:-fopenmp;-x;cuda>"
    INTERFACE_INCLUDE_DIRECTORIES "${KOKKOS_IMPORT_PREFIX}/include"
  )

When I want to use trilinos in another project and link to trilinos, which will pull in Kokkos::Kokkos, which adds the compile flags -x cuda. Guarding KOKKOS_ALL_COMPILE_OPTIONS works fine for me.

[dalg24-jenkins]

Can one of the admins verify this patch?

[dalg24]

Alternative might be to guard

kokkos/cmake/kokkos_arch.cmake

Lines 119 to 157 in 593597c

	#------------------------------- KOKKOS_CUDA_OPTIONS ---------------------------
	#clear anything that might be in the cache
	GLOBAL_SET(KOKKOS_CUDA_OPTIONS)
	# Construct the Makefile options
	IF (KOKKOS_ENABLE_CUDA_LAMBDA)
	IF(KOKKOS_CXX_COMPILER_ID STREQUAL NVIDIA)
	GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS "-expt-extended-lambda")
	IF(KOKKOS_COMPILER_CUDA_VERSION GREATER_EQUAL 110)
	GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS "-Wext-lambda-captures-this")
	ENDIF()
	ENDIF()
	ENDIF()
	IF (KOKKOS_ENABLE_CUDA_CONSTEXPR)
	IF(KOKKOS_CXX_COMPILER_ID STREQUAL NVIDIA)
	GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS "-expt-relaxed-constexpr")
	ENDIF()
	ENDIF()
	IF (KOKKOS_CXX_COMPILER_ID STREQUAL Clang)
	SET(CUDA_ARCH_FLAG "--cuda-gpu-arch")
	GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS -x cuda)
	IF (KOKKOS_ENABLE_CUDA)
	SET(KOKKOS_IMPL_CUDA_CLANG_WORKAROUND ON CACHE BOOL "enable CUDA Clang workarounds" FORCE)
	ENDIF()
	ELSEIF(KOKKOS_CXX_COMPILER_ID STREQUAL NVIDIA)
	SET(CUDA_ARCH_FLAG "-arch")
	ENDIF()
	IF (KOKKOS_CXX_COMPILER_ID STREQUAL NVIDIA)
	STRING(TOUPPER "${CMAKE_BUILD_TYPE}" _UPPERCASE_CMAKE_BUILD_TYPE)
	IF (KOKKOS_ENABLE_DEBUG OR _UPPERCASE_CMAKE_BUILD_TYPE STREQUAL "DEBUG")
	GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS -lineinfo)
	ENDIF()
	UNSET(_UPPERCASE_CMAKE_BUILD_TYPE)
	IF (KOKKOS_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 9.0 AND KOKKOS_CXX_COMPILER_VERSION VERSION_LESS 10.0)
	GLOBAL_APPEND(KOKKOS_CUDAFE_OPTIONS --diag_suppress=esa_on_defaulted_function_ignored)
	ENDIF()
	ENDIF()

The flag is added here

kokkos/cmake/kokkos_arch.cmake

Line 140 in 593597c

GLOBAL_APPEND(KOKKOS_CUDA_OPTIONS -x cuda)

Your solution looks fine to me but I want feedback from @jjwilke.
NOTE: Jenkins CI does not need to run for that PR.

[masterleinad]

This only affects Trilinos so guarding in the Trilinos-only part looks sensible to me and mirrors what we do for a regular build in

kokkos/cmake/kokkos_tribits.cmake

Lines 269 to 282 in 03ded4e

	IF (KOKKOS_ENABLE_CUDA)
	TARGET_COMPILE_OPTIONS(
	${LIBRARY_NAME}
	PUBLIC $<$<COMPILE_LANGUAGE:CXX>:${KOKKOS_CUDA_OPTIONS}>
	)
	SET(NODEDUP_CUDAFE_OPTIONS)
	FOREACH(OPT ${KOKKOS_CUDAFE_OPTIONS})
	LIST(APPEND NODEDUP_CUDAFE_OPTIONS -Xcudafe ${OPT})
	ENDFOREACH()
	TARGET_COMPILE_OPTIONS(
	${LIBRARY_NAME}
	PUBLIC $<$<COMPILE_LANGUAGE:CXX>:${NODEDUP_CUDAFE_OPTIONS}>
	)
	ENDIF()

.

[finkandreas]

But the problematic file is cmake/KokkosTrilinosConfig.cmake.in. This one, is a template and will ultimately install the file $INSTALLDIR/lib/cmake/kokkos/KokkosConfig.cmake. In this file the variable KOKKOS_ALL_COMPILE_OPTIONS is referenced. The compilation of trilinos works fine, but the installed cmake file is wrong.

[crtrott]

This bug is in Trilinos right now? I.e. its in the release?

[finkandreas]

I tested with Trilinos master branch, but looking at the source code of 13.0 release, it seems to be the same (did not try to compile it though)

[jjwilke]

This looks fine. This is working for nvcc because KOKKOS_CUDA_OPTIONS is empty. There is one spot with clang where we add a flag to KOKKOS_CUDA_OPTIONS even if CUDA isn't enabled. This is why we haven't seen the error before (i.e. no one builds downstream Trilinos projects with clang-cuda).

I'm not sure if we care about consistent behavior, i.e. should KOKKOS_CUDA_OPTIONS always be empty if CUDA is not enabled? Or should KOKKOS_CUDA_OPTIONS contain all the expected CUDA options even if CUDA is not enabled. Super picky, but nice to be clean and consistent.

[crtrott]

OK to test.

[dalg24]

Retest this please