silent performance degradation when cuda_ipc is not possible

[BenWibking]

Background information

There is a severe silent performance degradation (performance is reduced by 50% for my code!) when CUDA-aware MPI is being used but cuda_ipc has been disabled by the OpenMPI runtime.

The user is completely unaware that this is happening and is also unaware that a simple fix is available (do not set CUDA_VISIBLE_DEVICES and do not use managed memory). At the least, this should be documented. Ideally, a warning would be printed to stderr and an MPIX_ function would be available to query whether cuda_ipc is being used.

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

4.1.4 + UCX 1.12.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Installed via Spack on NCSA Delta

Please describe the system on which you are running

• Operating system/version: Rocky Linux 8.4
• Computer hardware: 1 64-core AMD EPYC 7763 Milan + 4 NVIDIA A100s w/ NVLink
• Network type: HPE/Cray Slingshot

---

Details of the problem

In principle, the performance degradation should be reproduced by running any GPU-aware MPI code. In my case, it can be reproduced by running this code: quokka-astro/quokka#121.