add sharded checkpoint loading for AutoTP path to reduce the peak mem… #3102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sywangyi
requested review from
RezaYazdaniAminabadi,
jeffra,
mrwyattii,
awan-10,
cmikeh2 and
arashb
as code owners
sywangyi
force-pushed
the
autoTP_memory_opt
branch
from
b48734a to
7340a93
Compare
…ory in initialization stage Signed-off-by: Wang, Yi A <[email protected]>
sywangyi
force-pushed
the
autoTP_memory_opt
branch
2 times, most recently
from
13dfd1c to
464db61
Compare
Signed-off-by: Wang, Yi A <[email protected]>
sywangyi
force-pushed
the
autoTP_memory_opt
branch
from
464db61 to
c40e911
Compare
|
Hi，@molly-smith, this PR is meant to reduce the host memory per Rank, support shard loading in AutoTP path, same with shard loading in kernel injection path. |
tjruwase
approved these changes
May 3, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ory in initialization stage