CUDA-Aware UCX with a mixture of CPU-only & GPU Nodes

[judicaelclair]

Some of the nodes in my cluster do not have a GPU, but I would like to use CUDA-aware UCX for OpenMPI / MPICH. Note that the cluster is equipped with a high-speed Mellanox Ethernet interconnect.

The only approach so far that seems to work is to build UCX twice, once for GPU nodes (with CUDA & gdrcopy), and another for CPU-only nodes (without CUDA nor gdrcopy). This allows for CPU<->GPU and GPU<->GPU communication between GPU nodes, but if I have GPU node A and CPU-only node B and I attempt to send a GPU buffer from A to B's host memory, UCX fails (i.e. CPU<->GPU doesn't work in this case). So,

• Is there a way to get this to work?
• And is building UCX twice as above an officially supported approach?

I've also tried using a single CUDA-aware UCX build for both GPU & CPU-only nodes, but this causes the CPU-only nodes to stall during initialisation.

Library versions:

• ucx: 1.17.0
• gdrcopy: 2.4.2