From 0441f07ac39390d8d2cf5c175e8b5e32c046e18a Mon Sep 17 00:00:00 2001 From: Jeff Squyres Date: Tue, 26 Nov 2024 14:37:51 -0500 Subject: [PATCH 1/3] mpirun.1.rst: fix some PRRTE typos Fix some "PRRTE" typos (where they should be "PRTE"), and add some clarification language. Signed-off-by: Jeff Squyres (cherry picked from commit 9e7441f856b8a1b2f646444ea226285c37aa619b) --- docs/man-openmpi/man1/mpirun.1.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/man-openmpi/man1/mpirun.1.rst b/docs/man-openmpi/man1/mpirun.1.rst index 93aac7e03f5..ed7f5dfd58b 100644 --- a/docs/man-openmpi/man1/mpirun.1.rst +++ b/docs/man-openmpi/man1/mpirun.1.rst @@ -532,11 +532,11 @@ generally useful to most Open MPI users: just before launch. * ``--launch-agent``: Name of the executable that is to be used to - start processes on the remote nodes. The default is ``PRRTEd``. This + start processes on the remote nodes. The default is ``prted``. This option can be used to test new daemon concepts, or to pass options back to the daemons without having mpirun itself see them. For - example, specifying a launch agent of ``PRRTEd -mca odls_base_verbose - 5`` allows the developer to ask the ``PRRTEd`` for debugging output + example, specifying a launch agent of ``prted --prtemca odls_base_verbose + 5`` allows the developer to ask the ``prted`` for debugging output without clutter from ``mpirun`` itself. * ``--report-state-on-timeout``: When paired with the ``--timeout`` @@ -1013,10 +1013,12 @@ MCA parameters can be set not only on the mpirun command line, but alternatively in a system or user ``mca-params.conf`` file or as environment variables, as described in the :ref:`Setting MCA Parameters `. These are MCA parameters for -the PRRTE runtime so the command line argument ``--PRRTEmca`` must be used to -pass the MCA parameter key/value pair. Alternatively, the MCA parameter key/ -value pair may be specific on the command line by prefixing the key with -``PRRTE_MCA_``. Some examples include: +the PRRTE runtime so the command line argument ``--prtemca`` +(yes, ``prte`` with a single ``r``, not two ``r``'s) must be used to +pass the MCA parameter key/value pair. Alternatively, the MCA parameter +key/value pair may be specific on the command line by prefixing the key with +``PRTE_MCA_`` (again, that is not a typo: ``PRTE`` not ``PRRTE``). +Some examples include: .. list-table:: :header-rows: 1 @@ -1662,7 +1664,7 @@ that job are designated "secondary" jobs): summary print statement. By default, the job will abort when any process terminates with -non-zero status. The MCA parameter ``--PRRTEmca state_base_error_non_zero_exit`` +non-zero status. The MCA parameter ``--prtemca state_base_error_non_zero_exit`` can be set to "false" (or "0") to cause Open MPI to not abort a job if one or more processes return a non-zero status. In that situation the Open MPI records and notes that processes exited with non-zero From 7f039ad280f4d1ea784894e46106b447f7cacbd2 Mon Sep 17 00:00:00 2001 From: Jeff Squyres Date: Tue, 26 Nov 2024 14:40:10 -0500 Subject: [PATCH 2/3] mpirun.1.rst: fix MPI_COMM_WORLD font Put MPI_COMM_WORLD in fixed-width font. Signed-off-by: Jeff Squyres (cherry picked from commit 724dd86b514f2e3905626b272fd2d5a9bd54937f) --- docs/man-openmpi/man1/mpirun.1.rst | 34 +++++++++++++++--------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/man-openmpi/man1/mpirun.1.rst b/docs/man-openmpi/man1/mpirun.1.rst index ed7f5dfd58b..c70cbb6b106 100644 --- a/docs/man-openmpi/man1/mpirun.1.rst +++ b/docs/man-openmpi/man1/mpirun.1.rst @@ -262,7 +262,7 @@ To map processes: * ``--bynode``: Launch processes one per node, cycling by node in a round-robin fashion. This spreads processes evenly among nodes and - assigns MPI_COMM_WORLD ranks in a round-robin, "by node" manner. + assigns ``MPI_COMM_WORLD`` ranks in a round-robin, "by node" manner. (deprecated in favor of ``--map-by node``) * ``--cpu-list ``: Comma-delimited list of processor IDs to @@ -272,7 +272,7 @@ To map processes: .. note:: You can run Run the hwloc ``lstopo(1)`` command to see a list of available cores and their logical IDs. -To order processes' ranks in MPI_COMM_WORLD: +To order processes' ranks in ``MPI_COMM_WORLD``: * ``--rank-by ``: Rank in round-robin fashion according to the specified mode, defaults to slot. Supported options include @@ -311,15 +311,15 @@ To manage standard I/O: specified filename. Any directories in the filename will automatically be created. Each output file will consist of ``filename.id``, where the ``id`` will be the processes' rank in - MPI_COMM_WORLD, left-filled with zero's for correct ordering in + ``MPI_COMM_WORLD``, left-filled with zero's for correct ordering in listings. A relative path value will be converted to an absolute path based on the cwd where mpirun is executed. Note that this will not work on environments where the file system on compute nodes differs from that where :ref:`mpirun(1) ` is executed. -* ``--stdin ``: The MPI_COMM_WORLD rank of the process that is - to receive stdin. The default is to forward stdin to MPI_COMM_WORLD +* ``--stdin ``: The ``MPI_COMM_WORLD`` rank of the process that is + to receive stdin. The default is to forward stdin to ``MPI_COMM_WORLD`` rank 0, but this option can be used to forward stdin to any process. It is also acceptable to specify none, indicating that no processes are to receive stdin. @@ -329,7 +329,7 @@ To manage standard I/O: * ``--tag-output``: Tag each line of output to stdout, stderr, and stddiag with ``[jobid, MCW_rank]`` indicating the process - jobid and MPI_COMM_WORLD rank of the process that generated the + jobid and ``MPI_COMM_WORLD`` rank of the process that generated the output, and the channel which generated it. * ``--timestamp-output``: Timestamp each line of output to stdout, @@ -342,7 +342,7 @@ To manage standard I/O: specified file. * ``--xterm ``: Display the output from the processes - identified by their MPI_COMM_WORLD ranks in separate xterm + identified by their ``MPI_COMM_WORLD`` ranks in separate xterm windows. The ranks are specified as a comma-separated list of ranges, with a ``-1`` indicating all. A separate window will be created for each specified process. @@ -716,7 +716,7 @@ options that describe mapping policies. Consider the same hostfile as above, again with ``-n 6``. The table below lists a few ``mpirun`` variations, and shows which -MPI_COMM_WORLD ranks end up on which node: +``MPI_COMM_WORLD`` ranks end up on which node: .. list-table:: :header-rows: 1 @@ -832,7 +832,7 @@ Open MPI employs a three-phase procedure for assigning process locations and ranks: #. **Mapping**: Assigns a default location to each process -#. **Ranking**: Assigns an MPI_COMM_WORLD rank value to each process +#. **Ranking**: Assigns an ``MPI_COMM_WORLD`` rank value to each process #. **Binding**: Constrains each process to run on specific processors The mapping step is used to assign a default location to each process @@ -864,7 +864,7 @@ gives you detailed control over process binding as well. Rankfiles are discussed :ref:`below `. The second phase focuses on the ranking of the process within the -job's MPI_COMM_WORLD. Open MPI separates this from the mapping +job's ``MPI_COMM_WORLD``. Open MPI separates this from the mapping procedure to allow more flexibility in the relative placement of MPI processes. This is best illustrated by considering the following cases where we used the ``--np 8 --map-by ppr:2:package --host aa:4,bb:4`` option: @@ -1168,7 +1168,7 @@ Rankfiles are text files that specify detailed information about how individual processes should be mapped to nodes, and to which processor(s) they should be bound. Each line of a rankfile specifies the location of one process (for MPI jobs, the process' "rank" refers -to its rank in MPI_COMM_WORLD). The general form of each line in the +to its rank in ``MPI_COMM_WORLD``). The general form of each line in the rankfile is: .. code:: @@ -1301,11 +1301,11 @@ Standard I/O retained, or removed? Open MPI directs UNIX standard input to ``/dev/null`` on all processes -except the MPI_COMM_WORLD rank 0 process. The MPI_COMM_WORLD rank 0 +except the ``MPI_COMM_WORLD`` rank 0 process. The ``MPI_COMM_WORLD`` rank 0 process inherits standard input from ``mpirun``. .. note:: The node that invoked ``mpirun`` need not be the same as the - node where the MPI_COMM_WORLD rank 0 process resides. Open + node where the ``MPI_COMM_WORLD`` rank 0 process resides. Open MPI handles the redirection of ``mpirun``'s standard input to the rank 0 process. @@ -1322,7 +1322,7 @@ example: shell$ mpirun -n 2 my_app < my_input > my_output -Note that in this example only the MPI_COMM_WORLD rank 0 process will +Note that in this example only the ``MPI_COMM_WORLD`` rank 0 process will receive the stream from ``my_input`` on stdin. The stdin on all the other nodes will be tied to ``/dev/null``. However, the stdout from all nodes will be collected into the ``my_output`` file. @@ -1645,7 +1645,7 @@ that job are designated "secondary" jobs): * If one or more processes in the primary job normally terminate with non-zero exit status, ``mpirun`` returns the exit status of the - process with the lowest MPI_COMM_WORLD rank to have a non-zero + process with the lowest ``MPI_COMM_WORLD`` rank to have a non-zero status. * If all processes in the primary job normally terminate with exit @@ -1653,7 +1653,7 @@ that job are designated "secondary" jobs): terminate with non-zero exit status, ``mpirun``: #. Returns the exit status of the process with the lowest - MPI_COMM_WORLD rank in the lowest jobid to have a non-zero + ``MPI_COMM_WORLD`` rank in the lowest jobid to have a non-zero status, and #. Outputs a message summarizing the exit status of the primary and all secondary jobs. @@ -1707,7 +1707,7 @@ processes exited before calling :ref:`MPI_FINALIZE(3) `. If an internal error occurred in mpirun, the corresponding error code is returned. In the event that one or more processes exit before calling :ref:`MPI_FINALIZE(3) `, the return value of -the MPI_COMM_WORLD rank of the process that mpirun first notices died +the ``MPI_COMM_WORLD`` rank of the process that mpirun first notices died before calling :ref:`MPI_FINALIZE(3) ` will be returned. Note that, in general, this will be the first process that died but is not guaranteed to be so. From 3828afff9ba3decb1786232c5722b5221433e182 Mon Sep 17 00:00:00 2001 From: Jeff Squyres Date: Tue, 26 Nov 2024 18:45:09 -0500 Subject: [PATCH 3/3] mpirun.1.rst: Describe --pmixmca and --prtemca Describe these options and recomend that users use them when passing PMIx and PRRTE MCA parameters, respectively. Signed-off-by: Jeff Squyres (cherry picked from commit 05f62058cf8ac66b43f3db898a1129529c8f43a6) --- docs/man-openmpi/man1/mpirun.1.rst | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/man-openmpi/man1/mpirun.1.rst b/docs/man-openmpi/man1/mpirun.1.rst index c70cbb6b106..4b20b874e36 100644 --- a/docs/man-openmpi/man1/mpirun.1.rst +++ b/docs/man-openmpi/man1/mpirun.1.rst @@ -422,7 +422,22 @@ Setting MCA parameters: * ``--mca ``: Send arguments to various MCA modules. See the :ref:`Setting MCA Parameters - ` section for mode details. + ` section for more details. + + .. note:: Open MPI will attempt to discern PMIx and PRRTE MCA + parameters passed via ``--mca`` and handle them + appropriately, but it may not always guess correctly. It + is best to use ``--pmixmca`` and ``--prtemca`` when + passing MCA parammeters to PMIx and PRRTE, respectively. + +* ``--pmixmca ``: Send arguments to MCA modules in the + PMIx subsystem. See the :ref:`Setting MCA Parameters + ` section for more details. + +* ``--prtemca ``: Send arguments to MCA modules in the + PMIx Reference Runtime Environment (PRRTE) subsystem. See the + :ref:`Setting MCA Parameters ` + section for more details. * ``--tune ``: Specify a tune file to set arguments for various MCA modules and environment variables. See the :ref:`