[WIP] Unified MPI for C / SST

[germasch]

So maybe this should have been part of the previous PR #1466, because it turned out I had to change the SMPI_ wrappers are generated a bit to make this all work from C as well...

This PR adds MPI wrapping support for plain C, and uses it in SST, both to get rid of sstmpidummy.h and to make SST work serially on a parallel build with MPI_Init.

I add some comments, too:

// Overview of the MPI wrapping:
// =============================
//
// Including this header provides a unified way of dealing with the
// building against an actual MPI library (ADIOS2_HAVE_MPI) vs using
// a dummy implementation, and also deals with running serially
// when built against MPI, but not calling MPI_Init
//
// MPI types / constants
// ---------------------
//
// If ADIOS2_HAVE_MPI is defined, the actual MPI types (in particular MPI_Comm)
// from the real MPI are always used, even in the dummy implementations that are
// used when MPI hasn't been initialized.
// Otherwise, the definitions from mpidummy.h are used. These types are in the
// global namespace.
//
// MPI_* functions
// -------------
//
// The usual MPI functions are always defined in the global namespace (extern
// "C"). If ADIOS2_HAVE_MPI is defined, the declarations come from mpi.h.
// Otherwise, these functions are declared in mpidummy.h, also extern "C".
//
// If ADIOS2_HAVE_MPI is defined, the declarations in mpidummy.h are put into
// adios2::helper::mpidummy in order to not interfere with the extern "C" ones,
// but this way they are still available to use in the SMPI_* wrappers
//
// SMPI_* wrappers
// ---------------
// The SMPI wrappers provide a way to write code that works in all three of the
// following cases:
// - If ADIOS2_HAVE_MPI is undefined, these wrappers call the corresponding
//   extern "C" MPI functions, which in this case are the mpidummy
//   implementations.
// - If ADIOS2_HAVE_MPI is defined, and MPI has been initialized, these wrappers
//   call the corresponding extern "C" MPI functions, which in this case are the
//   real MPI implementations.
// - If ADIOS2_HAVE_MPI is defined, and MPI has not been initialized, these wrappers
//   call the corresponding adios2::helper::mpidummy MPI functions, which are
//   serial only but don't require the real MPI to be initialized.

I guess I can't request reviewers, so I'll just mention @pnorbert @eisenhauer @williamfgc .

[germasch]

@pnorbert @eisenhauer: you might want to wait until I figured out why some of the tests are failing :(

[eisenhauer]

Looks good to me...

[germasch]

Alright, so it finally went through the CI, thanks to @eisenhauer for helping with debugging.

The CI found a real problem, though I couldn't reproduce it on my macbook: MPI_Allreduce was called with type MPI_LONG within SST. The mpidummy implementation uses MPI_Type_size, which didn't handle the MPI_LONG data type, but returned MPI_ERR_TYPE. That error got propagated through MPI_Reduce/MPI_Allreduce, but wasn't checked in SST, leading to the non-obvious failures.

The vast majority of MPI API calls are not error checked in ADIOS2. I think that's perfectly fine, because the default MPI behavior is MPI_ERRORS_ARE_FATAL, which means that the code will abort within MPI on error, before any potential error handling in ADIOS2 kicks in. mpidummy, however, did not have MPI_ERRORS_ARE_FATAL behavior, relying on the app to check error returns, instead which mostly doesn't happen. So I introduced a RETURN_CHECK macro into mpidummy that will abort the code on MPI errors within mpidummy, just as happens in a real MPI. alleviating the need for the rest of ADIOS2 to check return codes.

[pnorbert]

Looks good.
What is that do { } while(0) trick in the DISPATCH macro?

[germasch]

What is that do { } while(0) trick in the DISPATCH macro?

It's not strictly needed there, but it's a commonly used idiom to make a macro behave like a function, including forcing a semicolon after use, and making things correctly when used with control structures like if: https://www.pixelstech.net/article/1390482950-do-%7B-%7D-while-%280%29-in-macros

[pnorbert]

Oops, I merged without noticing the [WIP]. Should we revert this?

[germasch]

It's actually ready, I just forgot to edit the title again. If you don't mind the [WIP] in the merge commit message, it's fine.