sessions:deworldify behavior of pmix pset lookup

[hppritcha]

It turns out that the existing ompi_instance_group_pmix_pset implementation assumes an MPI_COMM_WORLD type of model.

This prevents the ability to use more dynamically generated process sets, possibly using an external agent.

Swith to using the pmix pset membership query to find new pset membership.

Related to #10862

Signed-off-by: Howard Pritchard [email protected]

[rhc54]

I don't see the backend implementation in PRRTE for this query, but I will add it ASAP. FWIW, I don't believe you will need to refresh the cache, but I will check and report back once I have it implemented.

[hppritcha]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[hppritcha]

could someone from nvidia
@janjust post what's going on with mellanox ci. I try to access the logs but azure gets borked up and never brings up the logs.

[janjust]

It worked for me. Attached raw log.
raw_azp_log.txt

[HawkmoonEternal]

There seems to be a related case in the ompi_instance_get_nth_pset function.
The current implementation only refreshes the process set names once in the beginning, thus not allowing for the discovery of dynamically added process sets later on:

 int ompi_instance_get_nth_pset (ompi_instance_t *instance, int n, int *len, char *pset_name)
{
    if (NULL == ompi_mpi_instance_pmix_psets && n >= ompi_instance_builtin_count) {
        ompi_instance_refresh_pmix_psets (PMIX_QUERY_PSET_NAMES);
    }
...

The condition could be changed to something like this:

if (NULL == ompi_mpi_instance_pmix_psets || (size_t) n >= (ompi_instance_builtin_count + ompi_mpi_instance_num_pmix_psets)) {

[rhc54]

I think this is where we are getting into terminology confusion. In PMIx, a "process set" is immutable and assigned for each application at time of execution, assuming the user provides a string "process set" name. Dynamic collections of procs are called "groups" in PMIx, and one can assemble any arbitrary collection of procs into a group.

Unfortunately, MPI chose to define "process set" to be the equivalent of a PMIx "group". Querying PMIx "pset_names" is only going to return the currently executing PMIx "process sets". This can change in a persistent DVM as more jobs are submitted, but it change due to any dynamic creation of MPI "process sets". For that info, you would have to query the current set of PMIx groups.

[hppritcha]

bot:aws:retest

[rhc54]

Can't speak to the OMPI code portions, but the PMIx queries look okay to me. Have you checked about the cache refresh yet? I don't remember if it was necessary or not.

Thanks for cleaning up the query on the PRRTE side of things!

[hppritcha]

@wenduwan please review when you have a chance

[wenduwan]

Running tests...

[hppritcha]

@wenduwan how did testing go?

[wenduwan]

Test passed a long time ago.. I forgot to come back.

[hppritcha]

@wenduwan now that 5.0.2rc1 is out could you review this PR?

[wenduwan]

I don't understand this change and its context, but it's so old that I think it's a good idea to merge and finally run it through MTT etc.