{lib}[GCC/11.2.0] MPItrampoline v2.8.0 w/ OpenMPI v4.1.2 (WIP) #14607
Conversation
boegel
changed the title
|
@boegelbot please test @ generoso |
|
@boegel: Request for testing this PR well received on login1 PR test command '
Test results coming soon (I hope)... - notification for comment with ID 999069507 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @boegelbot |
|
@boegel: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-easyconfigs/actions/runs/1608513698 bleep, bloop, I'm just a bot (boegelbot v20200716.01) |
|
In principle, it is possible to install MPItrampoline by itself, without any other MPI implementation. In practice, it can be convenient to ensure that a real MPI implementation together with its MPIwrapper are installed at the same time, since this provides a convenient fallback for users of MPItrampoline. It seems you want to do this here. When doing so, it is important that MPItrampoline is installed as a regular library (that can be found by applications), whereas the real MPI implementation as well as MPIwrapper must remain hidden. (Of course, MPIwrapper must be pointed to OpenMPI.) The "clients" of MPItrampoline must not accidentally see the real MPI library nor MPIwrapper. When MPItrampoline is configured, it might be convenient to add configuration options pointing it to the installed MPIwrapper, if you have enough control over the install path to make this possible. If so, the environment variable I don't know enough of EasyBuild to know how to achieve this. If you were using
This should get you started. I'd be happy to discuss further. In addition to the above, the environment variable For debugging, you can set the environment variable As briefly mentioned in the talk, on macOS, OpenMPI uses the build option find . -type f -print0 | xargs -0 perl -pi -e 's/-Wl,-flat_namespace//g';
./autogen.plFun fact: OpenMPI and MPIwrapper can be compiled with a completely different tool chain than MPItrampoline. (Of course, the CPU architecture must match.) Any incompatibilities between C++ and Fortran compilers will not matter. |
| default_component_specs = {'start_dir': '%(name)s-%(version)s'} | ||
|
|
||
| components = [ | ||
| ('MPIwrapper', '2.2.1', { |
There was a problem hiding this comment.
From the comment of @eschnett it seems that MPIwrapper (and OpenMPI which would be a dependency of it) should be extracted and used as a build dependency, that way we can reliably set the environment variables that MPItrampoline requires for runtime without having those modules loaded when something is being built with the toolchain.
There was a problem hiding this comment.
This would mean we would need to poke around in the module files that are generated for MPI implementations to make sure that required envvars are not being missed. Maybe the right thing to do is to have MPIwrapper (with OpenMPI as a build dependency) to be loaded but not allow it to add to LD_LIBRARY_PATH/LIBRARY_PATH?
There was a problem hiding this comment.
The MPIwrapper easyblock could be smart enough to inherit any MPI-relevant environment variables from OpenMPI (or whichever MPI it is wrapping).
There was a problem hiding this comment.
Technically, MPItrampoline loads MPIwrapper as a "plugin". (That's the technical term used by ld and/or cmake.) If you have another project using plugins, then you could look there for guidance.
|
@eschnett One question I was wondering about. Let's say a user wants to compile their own code, these use EB for some dependencies and then they load |
|
@ocaisa You cannot mix MPI libraries. This is like mixing MPICH and OpenMPI, or mixing OpenBLAS and MKL. I assume there would be linker errors, but if not, there will be a segfault at run time. Each application must choose a unique MPI implementation, and all its dependencies must also be built with the same MPI implementation. |
|
Ok, so from a user-facing perspective this would be a very major change, we would need a guard on the OpenMPI module that wouldn't allow it to be loaded when MPItrampoline is loaded. |
|
Isn't that the same case as with The whole point of |
|
Yes, indeed it's the same, the difference would be the brand recognition 😄 For end users who don't actually build their own software, EasyBuild can handle it all through it's toolchain definitions. Drilling that toolchain into people who do build their own software would be harder. In Lmod you can somehow handle this with the hierarchical module naming schemes and the |
|
@eschnett Sorry to keep bothering you, but I've thinking about the right way to implement this a lot. The issue I am concerned about is what happens if we arrange things such that the OpenMPI library can't be seen by the linker. In a typical scenario this means not loading the OpenMPI module file, which in turn means not loading the OpenMPI dependencies ( For example, here is what the loader resolution looks like for an OpenMPI from JSC: excluding loading the module would leave most of the libraries unresolved since |
|
I don't think that
My suggestion would be to ensure that When I build cmake -S . -B mpiwrapper-openmpi -DCMAKE_CXX_COMPILER=/cm/shared/apps/openmpi/gcc-9/64/4.1.0/bin/mpic++ -DCMAKE_Fortran_COMPILER=/cm/shared/apps/openmpi/gcc-9/64/4.1.0/bin/mpifort -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=$HOME/src/c/MPIstuff/mpiwrapper-openmpi
cmake --build mpiwrapper-openmpi
cmake --install mpiwrapper-openmpi |
|
@eschnett OK, but what if OpenMPI links to UCX? Those Are you implicitely assuming that the The default mechanism in EasyBuild is currently still to resolve libraries via |
|
MPItrampoline offers several options to "help" MPIwrapper load the underlying MPI library. The main problem in practice is that one doesn't have any control on how the underlying MPI library is installed on an HPC system. If you are building e.g. MPICH or OpenMPI yourself, then things are much easier. For example, in Julia we ship a standard MPICH for use with MPItrampoline, and this is built as a static library, and thus MPIwrapper is a single shared library (technically called a "plugin"). There are also various issues specific to macOS. I don't know whether this is a major concern for you. I am not aware of any macOS HPC system, so I didn't put much effort into this. Building MPICH or OpenMPI there should work fine. There is an environment variable I don't know of a way to influence the search path when loading the dependencies of a plugin (i.e. the actual MPI library). It would be straightforward to introduce a new environment variable You can set the environment variable In short:
As a side note: I tried renaming MPItrampoline's If you have further questions I'd be happy to have a Zoom call. |
|
Ok, I think we can probably work with The complexity is definitely in figuring out exactly what to do with MPIwrapper. We can break the effort into two parts, making the initial installation of MPItrampoline with an OpenMPI default (built as a component of that installation) should work already and we can already start exploring that. We can figure out the tricky part of MPIwrapper later. |
(created using
eb --new-pr)Marked as WIP, since I haven't experimented with this yet, and we need to figure out how to easily switch to other MPI libraries as well...
cc @eschnett