-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix missing deepspeed distributed call #9540
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Codecov Report
@@ Coverage Diff @@
## master #9540 +/- ##
=======================================
+ Coverage 89% 93% +4%
=======================================
Files 181 181
Lines 15159 15173 +14
=======================================
+ Hits 13469 14097 +628
+ Misses 1690 1076 -614 |
@SeanNaren @tchaton would you be in favor of making deepspeed plug-in inherit from ParallelPlugin directly? Flattening the inheritance structure reduces the risk of bugs like this |
What does this PR do?
Related #9521
After #9044 DeepSpeed was not updated to use the overridden function
init_ddp_connection
. I've refactored the code such that DeepSpeed handles the necessary initialisation.As noted by Adrian, #9044 was not included in 1.4.x, so this will be a fix for 1.5
Before submitting
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:
Did you have fun?
Make sure you had fun coding 🙃