Make G2G as default during the compilation...

[alazzaro]

.. and control it at runtime via DBCSR_USE_ACC_G2G env variable (OFF by default).

[alazzaro]

FYI @gsitaram

[hfp]

Quick Q, is G2G only hinging on GPU-aware MPI?

[alazzaro]

Quick Q, is G2G only hinging on GPU-aware MPI?

Yes, exactly.
The feature is now promoted to be "a runtime flag", but it is still experimental (there are few things to consider). It will be officially released in 2025...

[hfp]

Quick Q, is G2G only hinging on GPU-aware MPI?

Yes, exactly. The feature is now promoted to be "a runtime flag", but it is still experimental (there are few things to consider). It will be officially released in 2025...

Thanks!

I see __DBCSR_ACC_G2G also requires to calculate norms on GPU (to keep data in place). In general, I wonder if G2G would work in any case or if there are missing transfers. Without norms on GPU, I can think some transfers are missing, but is there anything else?

I consider implementing norms on GPU for OpenCL too. I think all contemporary MPIs have GPU support if say pointers are registered, etc.

[alazzaro]

I see __DBCSR_ACC_G2G also requires to calculate norms on GPU (to keep data in place). In general, I wonder if G2G would work in any case or if there are missing transfers. Without norms on GPU, I can think some transfers are missing, but is there anything else?

The short answer is: no, it would not work in any case, that's why it is still experimental. For this reason, I've added 31dc51a. Indeed, there can be cases where the kernels are too big (for instance, 50x50), so no kernel will be jitted and the library will fall-back to the CPU (without host data!) and fail...

The norms are only one part of the story (and only relevant when we do apply filtering). Another thing @gsitaram introduced: with G2G we move the B-transposed data, so we dont' need to run B-transpose for each step of the multiplication.

Overall, the speed-up on LUMI was quite significant for the H2O-DFT-LS...

[hfp]

Overall, the speed-up on LUMI was quite significant for the H2O-DFT-LS...

Good to know! I will try to get my way through it at some point.