-
Notifications
You must be signed in to change notification settings - Fork 27.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable multi-device for more models #30409
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this!
The necessity to move the values onto different devices makes me think that we want to specify more modules in _no_split_modules
rather than trying to handle like this.
It might not be possible in all cases, e.g. with the input_ids, but it shouldn't be necessary to do it for the attention modules
@amyeroberts Ready for review! All the models are supported using |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once again, a beautiful and impactful PR.
Thanks for all of your work on this and adding more capabilities to the library's models! ❤️
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Fixes #29786 (issue)
Who can review?
@amyeroberts