HIP and MPI+HIP updates

[ohearnk]

This MR ports the updated CUDA and MPI+CUDA codes (including recently merged f-function optimizations) to HIP and MPI+HIP versions, respectively. Also, this MR begins to unify the CUDA and HIP versions to simpily future GPU code maintainence.

Additional work also included in this PR concerns the following items:

• Integer-based atomic operation code paths (used in place of double precision atomics -mainly for older GPUs lacking hardware support [pre-pascal on NVIDIA GPUs])
• Wrappers around CUDA/HIP APIs (better error checking)
• HIP diagonalization (rocsolver re-enabled, CPU code disabled)
• Several bug fixed (memory leaks, initialization errors)
• ROCm version detection and disabling HIP builds against known buggy ROCm versions (>= v5.4.3)

Limitations / Known Issues:

• f function code (CMake flag -DENABLEF=TRUE) does not compile on HIP versions due to resource limitations (stack frame size exceeded in device functions). Errors are similar to the following (from a build on the AMD Accelerator Cloud platform):

error: stack frame size (143248) exceeds limit (131056) in function '_Z19getGrad_kernel_ffffPiPS_PdPS1_PP15HIP_vector_typeIiLj2EEPPhS7_S2_ii'
26 warnings and 1 error generated when compiling for gfx90a.
CMake Error at quick_hip_kernels_generated_gpu_get2e_grad_ffff.cu.o.Release.cmake:287 (message):
  Error generating file
  QUICK/build_ubuntu22_hip_gfx90a_rocm5.4.2_enablef_commit_0456378d/src/gpu/hip/CMakeFiles/quick_hip_kernels.dir//./quick_hip_kernels_generated_gpu_get2e_grad_ffff.cu.o

gmake[2]: *** [src/gpu/hip/CMakeFiles/quick_hip_kernels.dir/build.make:2080: src/gpu/hip/CMakeFiles/quick_hip_kernels.dir/quick_hip_kernels_generated_gpu_get2e_grad_ffff.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:570: src/gpu/hip/CMakeFiles/quick_hip_kernels.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2

Closes #344.

[ohearnk]

As noted above, all tests (full test suite) are passing for the CUDA and MPI+CUDA (1 GPU) versions. However, some tests are failing for the HIP and MPI+HIP versions. See the logs below from tests on the MI210s on the AMD AAC. Interestingly, the test failures are slightly different between the HIP and MPI+HIP versions.

This is a bit difficult to debug at least when comparing against the working HIP / MPI+HIP versions from the 23.08b release as there are also a number of test failures there. It may be better to pick a commit before the f-function optimizations and run tests there for comparison.

Test configuration on the AMD AAC:

• RHEL9 partition (1CN128C8G2H_2IB_MI210_RHEL9)
• ROCm v5.7.1, UCX v1.15.0, OpenMPI v4.1.6, GCC v11.3.1 (gfortran)
• f-function support disabled
• CMake configuration (HIP version):

cmake .. -DCOMPILER=MANUAL -DCMAKE_C_COMPILER=hipcc -DCMAKE_CXX_COMPILER=hipcc -DCMAKE_Fortran_COMPILER=gfortran -DMPI= -DHIP=TRUE -DQUICK_USER_ARCH=gfx90a -DENABLEF= -DCMAKE_INSTALL_PREFIX=${PWD}/../install_rhel9_hip_gfx90a_rocm5.7.1_ucx1.15.0_ompi4.1.6 -DHIP_TOOLKIT_ROOT_DIR=/shared/apps/rhel9/opt/rocm-5.7.1

HIP test summary and diffs:
runtest_hip.log
hip_test_diffs.log

MPI+HIP test summary and diffs:
runtest_mpi_hip_1gpu.log
mpi_hip_test_diffs.log

[agoetz]

I have tested git commit id ab27c99 on following platforms. I tested without f functions on delorean and chinotto and with f functions on Expanse A100 GPU nodes. All tests of the full test suite pass (serial, mpi, cuda, cudampi).

delorean

• AMD Ryzen 9 3900X 12-Core
• RTX 2080 Ti
• Almalinux 9.5
• cmake 3.26.5
• GNU 11.5.0 compiler
• OpenMPI 4.0.5
• CUDA 12.5

chinotto

• 2x 8-core Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
• 2x RTX 2080 Ti
• 1x Titan V
• CentOS 7.9.2009
• cmake 3.28.1
• GNU 10.2.1 compiler
• OpenMPI 4.0.5
• CUDA 12.0

Expanse

• 2x 20-core Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
• 4x A100 PCIe-40GB (2x used for testing)
• Rocky Linux 8.9
• cmake 3.28.1
• GNU 10.2.0 compiler
• OpenMPI 4.1.3
• CUDA 11.2.2

[agoetz]

Timers are not correct. It looks like 2e GRADIENT TIME is now added to EXC GRADIENT TIME.

Below outputs are for test QUICK-tests/benchmarks/input/morphine.in form the QUICK-tests repo executed on Expanse A100 GPU node.

QUICK-23.08a

------------- TIMING ---------------
| INITIAL GUESS TIME  =     1.127289064(  5.95%)
| DFT GRID OPERATIONS =     0.194754000(  1.03%)
| TOTAL LOAD BALANCING TIME =     0.007372800(  0.04%)
| TOTAL SCF TIME      =     6.730798000( 35.53%)
|       TOTAL OP TIME      =     6.292387000( 33.22%)
|             TOTAL 1e TIME      =     0.102761000(  0.54%)
|             TOTAL 2e TIME      =     2.693532000( 14.22%)
|             TOTAL EXC TIME     =     3.463376000( 18.28%)
|       TOTAL DII TIME      =     0.427670000(  2.26%)
|             TOTAL DIAG TIME    =     0.269553000(  1.42%)
| TOTAL GRADIENT TIME      =     8.303771000( 43.83%)
|       TOTAL 1e GRADIENT TIME      =     0.903336000( 4.82%)
|       TOTAL 2e GRADIENT TIME      =     5.969338000(31.51%)
|       TOTAL EXC GRADIENT TIME     =     1.421189000(  7.50%)
| TOTAL TIME          =    18.944405000
------------------------------------
| Job cpu time:  0 days  0 hours  0 minutes 18.9 seconds.

This PR 361, git commit id ab27c99

------------- TIMING ---------------
| INITIAL GUESS TIME  =     1.623547528(  8.93%)
| DFT GRID OPERATIONS =     0.137886000(  0.76%)
| TOTAL SCF TIME      =     6.646664000( 36.57%)
|       TOTAL OP TIME      =     6.225443000( 34.26%)
|             TOTAL 1e TIME      =     0.099403000(  0.55%)
|             TOTAL 2e TIME      =     2.631439000( 14.48%)
|             TOTAL EXC TIME     =     3.461705000( 19.05%)
|       TOTAL DII TIME      =     0.410465000(  2.26%)
|             TOTAL DIAG TIME    =     0.258074000(  1.42%)
| TOTAL GRADIENT TIME      =     8.619257000( 47.43%)
|       TOTAL 1e GRADIENT TIME      =     0.894716000( 4.98%)
|       TOTAL 2e GRADIENT TIME      =     0.000379000( 0.00%)     <-- This is wrong
|       TOTAL EXC GRADIENT TIME     =     7.714436000( 42.45%)    <-- This is wrong
| TOTAL TIME          =    18.172917000
------------------------------------