Support for opaque MPI wrapping

[olesenm]

Somewhere between an RFE or just a general discussion (@pnorbert @williamfgc or others).

As you may have seen, in OpenFOAM we hide away all of the MPI bits behind the Pstream library. The purpose of this is to have all of the MPI-specific bits hidden away in a separate library and no MPI directly included within the main code base. This allows swapping out of different MPI versions and/or MPI vendors without requiring a full recompilation of the code - just the MPI glue (takes a few minutes).
The time-savings are one aspect, but another distinct advantage is that we can quickly swap out say INTEL-MPI for OpenMPI to pinpoint differences or diagnose problems etc.

If we are looking to integrate ADIOS2 as part of OpenFOAM, it would be a definite advantage to have a similar way of hiding away the MPI dependency into a separate layer.

Since we'd obviously need to initialize ADIOS with an MPI communicator somewhere, one idea would be to define an opaque representation of it. We are doing something similar for managing lists of MPI_Request in an opaque manner with the UPstream::Request class being nothing better than an std::intptr_t content with some getter/setter methods. These are wrapped as shown in UPstreamWrapping.H to handle mpich variants (ints) or openmpi (pointers).

Could define something semantically similar for an opaque ADIOS Communicator. This would get us part of the way. The question remaining: how much of the other templated code could be reduced down to avoid direct use of MPI types?

Cheers,

/mark

[williamfgc]

@olesenm that's a great topic. One reason we didn't focus on such layer is because we rarely need to do this on our HPC systems. We rely on the vendor provided system-level MPI (IBM Spectrum on Summit or Cray-MPICH on Frontier). One workaround is what Ubuntu/Debian does for packaging, have separate MPICH and OpenMPI (and other vendors) compiled libraries. @bradking did awesome work to allow for MPI and non-MPI libraries in a single build to coexist a while ago, I don't know if that's the way to go.