Turn off xpmem in OFED 5.8 on Chrysalis #6359
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@amametjanov Can you check that this doesn't slow runs down? It did not on my test with an ne30 production coupled case. |
|
amametjanov
approved these changes
Apr 19, 2024
There was a problem hiding this comment.
Checked with 2 tests in
create_test e3sm_prod_bench
PFS.ne30pg2_r05_IcoswISC30E3r5.F2010.chrysalis_intel.bench-noio:
2024-04-18 22:53:17: MEMCOMP: Memory usage highwater changed by -0.16%: baseline=6373.210 MB, tolerance=5%, current=6362.930 MB
---------------------------------------------------
2024-04-18 22:53:17: TPUTCOMP: Throughput changed by 0.50%: baseline=1.791 sypd, tolerance=5%, current=1.782 sypd
PFS.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.bench-noio:
2024-04-18 22:52:00: MEMCOMP: Memory usage highwater changed by -0.45%: baseline=4902.090 MB, tolerance=5%, current=4879.850 MB
---------------------------------------------------
2024-04-18 22:52:00: TPUTCOMP: Throughput changed by 0.90%: baseline=1.997 sypd, tolerance=5%, current=1.979 sypd
<1% throughput tradeoff for <1% memory.
Turn off xpmem in OpenMPI Add env var to turn off xpmem when using OpenMPI. Avoids leaving nodes in unkillable state.
rljacob
force-pushed
the
rljacob/chrysalis/no-xpmem
branch
from
b757987 to
71db923
Compare
|
@amametjanov please try again with this new version that doesn't have the typo. |
|
PFS.ne30pg2_r05_IcoswISC30E3r5.F2010.chrysalis_intel.bench-noio: PFS.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.bench-noio: |
|
This is now in the openmpi module by default so don't need to add it. |
|
Removed it from module. Was in place 2pm to 10pm April 22. |
rljacob
added a commit
that referenced
this pull request
Apr 30, 2024
Add env var to chrysalis to turn off xpmem when using OpenMPI. Avoids leaving nodes in unkillable state. Workaround for bug in xpmem. [BFB]
rljacob
changed the title
|
revised title and comment because this variable is needed all the time, not just OpenMPI. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add env var to chrysalis to turn off xpmem when using the new OFED 5.8 network drivers.
A bug in xpmem can leave nodes stuck in an unkillable state after a model crash.
[BFB]