Neko on NVIDIA GPUs

[njansson]

To compile Neko for NVIDIA GPUs

• Make sure you have the CUDA Toolkit installed (e.g. nvidia-cuda-toolkit)
• Configure Neko to use CUDA using the --with-cuda=/path/to/cuda argument to configure, e.g.:

./configure  --with-cuda=/usr/local/cuda

• CUDA compiler flags and options can be passed using CUDA_CFLAGS, CUDA_ARCH and NVCC respectively, e.g:

./configure  --with-cuda=/usr/local/cuda CUDA_CFLAGS=-O3  CUDA_ARCH=-arch=sm_80 NVCC=/usr/local/cuda/bin/nvcc

• Build using make && make install
• Run in the same way as for a CPU build .e.g ./neko tgv.case, if everything went well backend type should be reported as CUDA, and the type of the device should be visible.

    _  __  ____  __ __  ____ 
   / |/ / / __/ / //_/ / __ \
  /    / / _/  / ,<   / /_/ /
 /_/|_/ /___/ /_/|_|  \____/ 
 
 (version: 0.3.99)
 (build: 2022-05-21 on x86_64-pc-linux-gnu using gnu)
 
  
 -------Job Information--------  
 Start time: 19:53 / 2022-06-01
 Running on: 1 MPI rank
 Bcknd type: Accelerator (CUDA)
 Dev. name : A100-PCIE-40GB
 Real type : double precision

[timofeymukha]

On kebnekaise ml fosscuda/2020b will pull in the necessary deps:

Currently Loaded Modules:
  1) snicenvironment (S)   4) zlib/1.2.11     7) CUDAcore/11.1.1  10) XZ/5.2.5           13) hwloc/2.2.0      16) GDRCopy/2.1-CUDA-11.1.1  19) PMIx/3.1.5       22) FFTW/3.3.8
  2) systemdefault   (S)   5) binutils/2.35   8) CUDA/11.1.1      11) libxml2/2.9.10     14) libevent/2.1.12  17) UCX/1.9.0-CUDA-11.1.1    20) OpenMPI/4.0.5    23) ScaLAPACK/2.1.0
  3) GCCcore/10.2.0        6) GCC/10.2.0      9) numactl/2.0.13   12) libpciaccess/0.16  15) Check/0.15.2     18) libfabric/1.11.0         21) OpenBLAS/0.3.12  24) fosscuda/2020b

[njansson]

Also, if OpenMPI or MPICH has been compiled with CUDA support, pass --enable-device-mpi to configure to enable (faster) device aware MPI.