MPI library used is the one compiled with the application and not the one used for mpirun. #11269
Comments
|
I think the problem is that The fix likely is in the PRRTE project - basically, omitting the "prefix" settings from those passed down to the application. Unfortunately, you might have a case where that needs to be done, so perhaps the correct answer is to define a new "application" prefix that is passed specifically down to the app and ignored by PRRTE? Or maybe the other way around if that helps with backward compatibility. |
|
you must also make sure Open MPI was built with |
|
This docs section may be helpful: https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/configure-cli-options/rpath-and-runpath.html |
|
William should go verify that we did default to using runpath, but I'm pretty sure we did. The larger issue is that when we rewrote the mpirun executable to deal with prrte, we didn't appear to add any of the code to have PRRTE set the LD_LIBRARY_PATH for applications when prefix is set (either explicitly or, in this case, because full path to mpirun was given), which means that that the linker is falling back to the runpath. |
|
Looks like it's defaulted to rpath: |
|
That's... not awesome. Can you include your config.log? |
|
From the compile command in config.log Do you need the entire log? |
|
Yes, or at least the part when it is figuring out runpath vs. rpath. And probably throw in |
|
|
@rhc54 Hi Ralph, reading our mpirun shim code, it looks like we set the local libdir as an env used by prrte: I can confirm that this libdir is the libdir of the mpirun, but it looks like LD_LIBRARY_PATH does not match on the compute nodes. Is PRRTE supposed to be using this env to set LD_LIBRARY_PATH or is there another way to propagate this information to PRRTE? |
|
I think that was supposed to be added (by OMPI developers, not Ralph), to the schizo framework. Maybe it never did? |
|
I see it there: $ ack OMPI_LIBDIR_LOC ~/pmix/prrte
/Users/rhc/pmix/prrte/src/mca/schizo/ompi/schizo_ompi.c
289: ompi_install_dirs_libdir = getenv("OMPI_LIBDIR_LOC");
$and then I see it used to deal with jar files: 251:static char *ompi_install_dirs_libdir = NULL;
263: asprintf(&str, fmt, app->app.argv[index], ompi_install_dirs_libdir, jarfile);
289: ompi_install_dirs_libdir = getenv("OMPI_LIBDIR_LOC");
290: if (NULL == ompi_install_dirs_libdir) {
310: if (NULL == strstr(app->app.argv[i], ompi_install_dirs_libdir)) {
313: asprintf(&value, "-Djava.library.path=%s%s", dptr, ompi_install_dirs_libdir);
315: asprintf(&value, "-Djava.library.path=%s:%s", dptr, ompi_install_dirs_libdir);
326: asprintf(&value, "-Djava.library.path=%s", ompi_install_dirs_libdir);
342: value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
364: value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
388: value = pmix_os_path(false, ompi_install_dirs_libdir, "mpi.jar", NULL);
$but nothing else. |
|
It looks like this was added purely for java support: I'll see if this can be extended to also extend the LD_LIBRARY_PATH. I see PRRTE has some code for this in plm_ssh_module.c, should the ompi schizo component be using the pass_libpath mca var to propogate this? |
|
You don't want to use an MCA param - we are discouraging those in PRRTE. Since this is a setting solely for the app, you probably want to simply add a pmix_envar_t to the app description that directs PRRTE to prepend the value to the existing envar. I'll take care of it and point you to it as an example of how to do these things. |
|
You know, just thinking about it, I don't think this will solve the problem you encountered. What this will do is set the library path so that whatever PRRTE launches will "see" the OMPI libraries associated with the What you wanted was to ensure that the OMPI job being launched used the MPI libraries it was built against - and not the ones associated with the build of So this seems to actually accomplish the opposite of what you seek - yes? |
|
I think the idea was that if you were using a specific mpirun to launch a job, you would want that MPI library to be used, @jsquyres @bwbarrett do you have opinions on what the exact behavior we are looking for should be? |
|
I personally don't care which direction you go, but please note that there was a lengthy debate about this before settling on the current behavior (I forget if it was on some issue ticket or the devel mailing list). I believe @devreal was one of those advocating for the current behavior (don't have I'd rather not get caught in a yo-yo situation, so perhaps you folks should hammer out what you want? Maybe controlled by some kind of option? |
|
In today's call, @wckzhang agreed to update this issue with a consensus of what is desired for v5.0.0. Thanks! |
|
We talked about this issue in the OMPI weekly telecon today and here's the summary of our consensus: We do not think the current behavior is intuitive. If the user specifies --prefix (or specifies the absolute path), we think that the LD_LIBRARY_PATH on the remote node should be set to include the mpirun's libmpi before invoking the application but after invoking the prted. This is the current documentation of prefix behavior, but it should be changed: |
|
@rhc54 I just realized that the behavior of setting LD_LIBRARY_PATH is different when you run mpirun on my head node vs a compute node, when I launch mpirun on a head node, it does populate my LD_LIBRARY_PATH But when launching mpirun on a compute node: That behavior seems a bit unexpected, is this intentional behavior on prrte's end? |
|
I'm unaware of any reason for that difference, nor what in the code would cause it. I'd take a hard look at the environment on each of those nodes to see if that envar is already set there, or being set by some |
|
Is one launching using slurm and another ssh? |
|
What I've realized from digging into it is that my Open MPI installations' libmpi.so are all configured with RPATH and not RUNPATH. This is despite not specifying their behavior, thus representing the default. When I directly specify "LDFLAGS=-Wl,--enable-new-dtags", it properly uses RUNPATH, and the linker does support runpath: I am currently digging into why the default is not RUNPATH. |
|
It looks like the wrapper compilers are doing the right thing and adding --enable-new-dtags. So the problem looks to be just libmpi.so is not being compiled with --enable-new-dtags. |
|
Isn't this behavior what we want? If I thought we want the executable (like |
|
I agree with Wei - intended behavior is that the libraries produced by MPI ( But for the user's application (the thing compiled with mpicc), it should be runpathed if runpath is supported. So reading your last update, I'm confused, if you look at the executable you're trying to run, is it reporting RUNPATH or RPATH? |
|
The behavior I see is: When I compile libmpi without RUNPATH, and then I change LD_LIBRARY_PATH, the path of libmpi shown under ldd <osu_binary> does not change to the LD_LIBRARY_PATH. However, when I do compile libmpi with RUNPATH, it does change. This behavior seems tied to the way libmpi was compiled as when I change LD_LIBRARY_PATH for libfabric libraries, it changes regardless of how I compiled Open MPI. Now to me, this doesn't seem to be the behavior we want because libmpi does not change when using LD_LIBRARY_PATH, but maybe my understanding is flawed. |
|
I'm going to write some simplified libraries and applications and try to nail down the exact behavior of RUNPATH/RPATH due to the confusion. |
|
I think potentially the source of confusion is that RUNPATH was used and the LD_LIBRARY_PATH was searched, but it was not detecting libmpi.so.80 (because libmpi.so.40 was in the path only) and so it went to RUNPATH, should be simple enough to clear up |
|
That indeed has been the issue. If I compile two different installations of ompi 5.0.x with default behavior, it doesn't have an issue switching between the two (confirmed that the readelf of libmpi.so was compiled with RPATH). So the root cause of this is that the search path is looking for a newer version of the libmpi library. As this is actually the behavior we want and the actual wrapper compilers do modify the applications behavior to RUNPATH, I think everything works and there isn't a problem. I have also understood why this behavior occurs: This may be another bug, but it appears LD_LIBRARY_PATH is NOT set when you're running locally. It worked as expected which was the main source of confusion. I also confirmed that running the binary itself switched between the libraries. To sum it up:
|
|
I don't think that prefix value being ignored locally is unintended behavior either based on the man page: It specifically calls out "remote node", and thus I don't see any issues here (though we should fix this prefix doc since there are some inaccuracies - before Open MPI is incorrect, it's before the application). I will resolve this issue |
What about the following scenario: application compiled against libmpi.so.40, Would the intention be for the application to use |
|
I tested this just now: It does indeed search the LD_LIBRARY_PATH first, but it is ONLY looking for libmpi.so.40 and not libmpi.so.80. |
|
I take it this means that an application compiled against 4.1.x will not be able to run if there is only 5.0.x available on the system. Is that intended? |
It depends on what you want that There is the There is the IF you are using the internal You need a way to tell |
Moved ticket from - openpmix/openpmix#2898
Summary:
I compiled OMB against both Open MPI 5.0.x and 4.1.4. Since no ABI break should have occurred, I expected to be able to run the 5.0.x mpirun to launch the 4.1.4 compiled OMB. However, this did not work. The root cause of the issue is that the MPI library used will be the one the application compiles against. I'm not convinced this is the behavior that we want to have.
The text was updated successfully, but these errors were encountered: