stream: implement gpu stream enqueue functions #5906
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
hzhou
force-pushed
the
2203_stream_2
branch
11 times, most recently
from
b8150a8 to
f4a5202
Compare
|
test:mpich/ch4/gpu/ofi ✔️ |
hzhou
force-pushed
the
2203_stream_2
branch
3 times, most recently
from
3289032 to
2501347
Compare
|
test:mpich/ch4/gpu/ofi ✔️ |
hzhou
force-pushed
the
2203_stream_2
branch
3 times, most recently
from
4d468f7 to
00ebf2b
Compare
|
Fixing the |
|
test:mpich/ch3/tcp |
1 similar comment
|
test:mpich/ch3/tcp |
raffenet
previously approved these changes
Apr 4, 2022
hzhou
changed the title
|
test:mpich/ch4/ofi ✔️ |
hzhou
force-pushed
the
2203_stream_2
branch
6 times, most recently
from
f7bedc1 to
1664db3
Compare
|
test:mpich/ch4/gpu/ofi |
|
test:mpich/ch4/most |
|
test:mpich/ch3/most |
|
test:mpich/ch4/gpu/ofi |
|
test:mpich/ch4/gpu/ofi |
|
test:mpich/ch4/ofi |
|
test:mpich/ch4/ofi |
raffenet
approved these changes
Apr 27, 2022
We only implemented the cuda version for now, which simply calls cudaLaunchHostFunc.
Add a wrapper to validate the raw gpu stream value so we can alert the user as early as we can.
Validate the gpu stream value right away in MPIX_Stream_create. If we go ahead with an invalid value, it will segfault later in cudaAPI in a rather mysterious way.
Add MPIR_Typerep_pack_stream and MPIR_Typerep_unpack_stream as wrappers for yaksa_pack_stream and yaksa_unpack_stream.
Only supported if the communicator has a cuda stream associated. Enqueues operation to the cuda stream.
It is often not critical for gpu streams to use separate vcis as long it is separated from normal traffic. In addition, it can be common for an application to use many GPU streams globally and we may not have enough dedicated vcis to match them. This commit default to reuse a single dedicated vci for gpu streams.
It tests the CUDA stream enqueue functions via MPIX Stream Communicator.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Implementing the gpu stream aware extension using the MPI Stream concept as discussed in #5908. This is a replacement implementation of #5905
Reference discussions on
MPIX_Stream: #5908EDIT:
Rebased and updated on top of #5946
[skip warnings]
Notes
The enqueue operation will work as long as there is no immediate GPU communication between the enqueue and synchronize. I assume this is the common case for applications. A proper implementation need isolate the traffic. We will do that after pt2pt: add MPID_Allocate_vci #5904
We need a way to pass binary type as info string. Maybe --
MPIX_Info_set_hex(info, key, &stream, sizeof(stream))and a corresponding getter.
The current
MPIR_Infohas aNULL-key as first entry. Need fix. -- ref: PR info: use dynamic array in MPIR_Info #5922Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.