ch4/ofi: Add psm3 capability set #5864

nitbhat · 2022-02-23T16:58:02Z

Pull Request Description

This PR adds a capability set for the PSM3 provider. All of the current configuration is borrowed from the PSM2 provider.

Author Checklist

Provide Description
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits Follow Good Practice
Commits are self-contained and do not do two things at once.
Commit message is of the form: module: short description
Commit message explains what's in the commit.
Passes All Tests
Whitespace checker. Warnings test. Additional tests via comments.
Contribution Agreement
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.

hzhou · 2022-02-23T20:43:49Z

We disabled libfabric psm3 --

mpich/src/mpid/ch4/netmod/ofi/subconfigure.m4

Line 276 in ebe7e0a

ofi_subdir_args="--enable-embedded --disable-psm3"

-- due to compile issues on our Jenkins. We need to revisit the issue and see if it can be resolved.

nitbhat · 2022-04-05T19:27:37Z

@hzhou, shall I enable psm3 and test this on jenkins to see what compile issues you're getting?

hzhou · 2022-04-05T19:28:49Z

@hzhou, shall I enable psm3 and test this on jenkins to see what compile issues you're getting?

Sure. Go ahead

hzhou · 2022-04-05T19:33:11Z

@nitbhat When you done, use the custom test for it. For example -- #5904 (comment)

The default review tests will use pre-built modules, which skips the rebuild of libfabric.

nitbhat · 2022-04-05T21:09:47Z

test:mpich/custom
netmod: ch4:ofi

nitbhat · 2022-04-05T21:41:18Z

test:mpich/custom
netmod: ch4:ofi

hzhou · 2022-04-05T22:24:59Z

Looks like it is working now. For reference, the original reason we disabled psm3 is here -- #5344 (comment)

nitbhat · 2022-04-06T13:26:55Z

@hzhou: The build looks good and the tests seem to be working. But, is there a way I can specify to the test framework on jenkins to use the psm3 provider? (maybe with netmod:ch4:ofi:psm3?)

I see that sockets is used as the provider for the tests.

hzhou · 2022-04-06T13:52:15Z

@hzhou: The build looks good and the tests seem to be working. But, is there a way I can specify to the test framework on jenkins to use the psm3 provider? (maybe with netmod:ch4:ofi:psm3?)

I see that sockets is used as the provider for the tests.

Our Jenkins only have Infiniband card. Does psm3 work with that?

nitbhat · 2022-04-06T13:54:57Z

Yes, psm3 works with Infiniband, it is a performant alternative to sockets over IB.

hzhou · 2022-04-06T13:58:17Z

test:mpich/custom
netmod: ch4:ofi
env: FI_PROVIDER=psm3

hzhou · 2022-04-06T13:59:29Z

test:mpich/ch4/ofi

nitbhat · 2022-04-06T14:00:50Z

Currently, the PSM3 provider also needs another environment variable to work correctly, PSM3_MULTI_EP=1. I'll relaunch the tests with that if the tests failing, which I'm guessing will.

nitbhat · 2022-04-21T21:17:29Z

test:mpich/custom
netmod: ch4:ofi
env: FI_PROVIDER=psm3

nitbhat · 2022-04-27T15:01:27Z

There is only one failure seen for psm3. I have seen this issue before and it happens intermittently for some tests. I've opened #5975 and will add an xfail to this PR.

nitbhat · 2022-04-27T15:07:57Z

test/mpi/maint/jenkins/xfail.conf

@@ -86,6 +86,9 @@
 # Sunf90 forbid passing cray pointer as integer
 * solstudio * * *       /^allocmemf90/          xfail=ticket0      f90/ext/testlist

+# psm3 specific failures
+* * * ch4:ofi:psm3 *    /^p_red .*/                      xfail=issue5975        coll/testlist


@hzhou: Is the right way to xfail a psm3 specific failure?

hzhou · 2022-04-27T15:08:08Z

There is only one failure seen for psm3. I have seen this issue before and it happens intermittently for some tests. I've opened #5975 and will add an xfail to this PR.

The xfail entry won't work against the custom tests since psm3 is selected via FI_PROVIDER, not configure device option. Since we don't have review tests targeting psm3 yet, the xfail entries is not necessary. Once we add such jobs, we can add xfail entries targeting the particular job or jenkins config option. Hopefully, the bug is fixed by then 🤞

hzhou · 2022-04-27T15:08:49Z

test:mpich/custom
netmod: ch4:ofi

nitbhat · 2022-04-27T15:18:41Z

There is only one failure seen for psm3. I have seen this issue before and it happens intermittently for some tests. I've opened #5975 and will add an xfail to this PR.

The xfail entry won't work against the custom tests since psm3 is selected via FI_PROVIDER, not configure device option. Since we don't have review tests targeting psm3 yet, the xfail entries is not necessary. Once we add such jobs, we can add xfail entries targeting the particular job or jenkins config option. Hopefully, the bug is fixed by then 🤞

Okay, understood. I'll go ahead and remove that commit in that case.

nitbhat · 2022-04-27T15:19:27Z

test:mpich/custom
netmod: ch4:ofi

hzhou · 2022-04-27T15:23:11Z

Since psm3 works on infiniband, we may consider using psm3 as default for testing. We'll need fix the issue before the switch.

hzhou · 2022-04-27T15:23:59Z

@nitbhat Does psm3 make psm2 obsolete?

nitbhat · 2022-04-28T17:11:09Z

@nitbhat Does psm3 make psm2 obsolete?

@hzhou: No, psm2 continues to be the preferred provider for OPA.

hzhou

LGTM

hzhou · 2022-04-29T17:16:11Z

@nitbhat Approved. Could you rebase?

nitbhat · 2022-04-29T19:34:07Z

@nitbhat Approved. Could you rebase?

Yes, rebased.

hzhou · 2022-04-29T19:43:32Z

test:mpich/warnings

nitbhat changed the title ~~Add psm3 capability set~~ ch4: Add psm3 capability set Feb 23, 2022

nitbhat changed the title ~~ch4: Add psm3 capability set~~ ch4/ofi: Add psm3 capability set Feb 23, 2022

nitbhat force-pushed the add_psm3_cap_set branch from cec1284 to faab8e6 Compare April 5, 2022 21:07

nitbhat force-pushed the add_psm3_cap_set branch from faab8e6 to 99baefb Compare April 5, 2022 21:40

nitbhat mentioned this pull request Apr 27, 2022

ch4/ofi: p_red times out with psm3 #5975

Closed

nitbhat force-pushed the add_psm3_cap_set branch 2 times, most recently from 7fdb3f0 to 2e042ac Compare April 27, 2022 15:06

nitbhat commented Apr 27, 2022

View reviewed changes

nitbhat force-pushed the add_psm3_cap_set branch from 2e042ac to 4b6f1ef Compare April 27, 2022 15:19

hzhou approved these changes Apr 29, 2022

View reviewed changes

nitbhat added 2 commits April 29, 2022 14:33

ch4/ofi: Create PSM3 Capability set

e785543

ch4/ofi: Add psm3 to subconfigure

fa162c4

nitbhat force-pushed the add_psm3_cap_set branch from 4b6f1ef to fa162c4 Compare April 29, 2022 19:33

hzhou merged commit 6b09399 into pmodels:main Apr 29, 2022

hzhou mentioned this pull request May 4, 2022

ch4/ofi: libfabric psm3 causing segfaults #5987

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ch4/ofi: Add psm3 capability set #5864

ch4/ofi: Add psm3 capability set #5864

nitbhat commented Feb 23, 2022 •

edited by hzhou

Loading

hzhou commented Feb 23, 2022

nitbhat commented Apr 5, 2022

hzhou commented Apr 5, 2022

hzhou commented Apr 5, 2022

nitbhat commented Apr 5, 2022

nitbhat commented Apr 5, 2022

hzhou commented Apr 5, 2022

nitbhat commented Apr 6, 2022 •

edited

Loading

hzhou commented Apr 6, 2022

nitbhat commented Apr 6, 2022

hzhou commented Apr 6, 2022

hzhou commented Apr 6, 2022

nitbhat commented Apr 6, 2022

nitbhat commented Apr 21, 2022

nitbhat commented Apr 27, 2022

nitbhat Apr 27, 2022

hzhou commented Apr 27, 2022

hzhou commented Apr 27, 2022

nitbhat commented Apr 27, 2022

nitbhat commented Apr 27, 2022

hzhou commented Apr 27, 2022

hzhou commented Apr 27, 2022 •

edited

Loading

nitbhat commented Apr 28, 2022

hzhou left a comment

hzhou commented Apr 29, 2022

nitbhat commented Apr 29, 2022 •

edited

Loading

hzhou commented Apr 29, 2022

ch4/ofi: Add psm3 capability set #5864

ch4/ofi: Add psm3 capability set #5864

Conversation

nitbhat commented Feb 23, 2022 • edited by hzhou Loading

Pull Request Description

Author Checklist

hzhou commented Feb 23, 2022

nitbhat commented Apr 5, 2022

hzhou commented Apr 5, 2022

hzhou commented Apr 5, 2022

nitbhat commented Apr 5, 2022

nitbhat commented Apr 5, 2022

hzhou commented Apr 5, 2022

nitbhat commented Apr 6, 2022 • edited Loading

hzhou commented Apr 6, 2022

nitbhat commented Apr 6, 2022

hzhou commented Apr 6, 2022

hzhou commented Apr 6, 2022

nitbhat commented Apr 6, 2022

nitbhat commented Apr 21, 2022

nitbhat commented Apr 27, 2022

nitbhat Apr 27, 2022

Choose a reason for hiding this comment

hzhou commented Apr 27, 2022

hzhou commented Apr 27, 2022

nitbhat commented Apr 27, 2022

nitbhat commented Apr 27, 2022

hzhou commented Apr 27, 2022

hzhou commented Apr 27, 2022 • edited Loading

nitbhat commented Apr 28, 2022

hzhou left a comment

Choose a reason for hiding this comment

hzhou commented Apr 29, 2022

nitbhat commented Apr 29, 2022 • edited Loading

hzhou commented Apr 29, 2022

nitbhat commented Feb 23, 2022 •

edited by hzhou

Loading

nitbhat commented Apr 6, 2022 •

edited

Loading

hzhou commented Apr 27, 2022 •

edited

Loading

nitbhat commented Apr 29, 2022 •

edited

Loading