Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable multi-device for more models #30409

Merged
merged 41 commits into from
Apr 30, 2024
Merged

Conversation

jla524
Copy link
Contributor

@jla524 jla524 commented Apr 23, 2024

What does this PR do?

Fixes #29786 (issue)

$ pytest tests/models/align/test_modeling_align.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 268 items / 247 deselected / 21 selected                                                                                         

tests/models/align/test_modeling_align.py ....................s                                                                      [100%]

<warnings redacted>
======================================== 20 passed, 1 skipped, 247 deselected, 2 warnings in 5.03s =========================================
$ pytest tests/models/altclip/test_modeling_altclip.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 267 items / 246 deselected / 21 selected                                                                                         

tests/models/altclip/test_modeling_altclip.py .....................                                                                  [100%]

<warnings redacted>
============================================== 21 passed, 246 deselected, 3 warnings in 3.32s ==============================================
$ pytest tests/models/bert/test_modeling_bert.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 158 items / 150 deselected / 8 selected                                                                                          

tests/models/bert/test_modeling_bert.py ........                                                                                     [100%]

<warnings redacted>
============================================== 8 passed, 150 deselected, 2 warnings in 4.29s ===============================================
$ pytest tests/models/blip/test_modeling_blip_text.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 80 items / 73 deselected / 7 selected                                                                                            

tests/models/blip/test_modeling_blip_text.py .......                                                                                 [100%]

<warnings redacted>
=============================================== 7 passed, 73 deselected, 1 warning in 2.69s ================================================
$ pytest tests/models/chinese_clip/test_modeling_chinese_clip.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 270 items / 249 deselected / 21 selected                                                                                         

tests/models/chinese_clip/test_modeling_chinese_clip.py .....................                                                        [100%]

<warnings redacted>
============================================== 21 passed, 249 deselected, 3 warnings in 6.57s ==============================================
$ pytest tests/models/depth_anything/test_modeling_depth_anything.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 109 items / 102 deselected / 7 selected                                                                                          

tests/models/depth_anything/test_modeling_depth_anything.py .......                                                                  [100%]

<warnings redacted>
============================================== 7 passed, 102 deselected, 2 warnings in 5.10s ===============================================
$ pytest tests/models/dinov2/test_modeling_dinov2.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 119 items / 112 deselected / 7 selected                                                                                          

tests/models/dinov2/test_modeling_dinov2.py .......                                                                                  [100%]

<warnings redacted>
============================================== 7 passed, 112 deselected, 2 warnings in 7.56s ===============================================
$ pytest tests/models/dinov2/test_modeling_dinov2.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 119 items / 112 deselected / 7 selected                                                                                          

tests/models/dinov2/test_modeling_dinov2.py .......                                                                                  [100%]

<warnings redacted>
============================================== 7 passed, 112 deselected, 2 warnings in 7.44s ===============================================
$ pytest tests/models/efficientformer/test_modeling_efficientformer.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 112 items / 105 deselected / 7 selected                                                                                          

tests/models/efficientformer/test_modeling_efficientformer.py .......                                                                [100%]

<warnings redacted>
============================================== 7 passed, 105 deselected, 2 warnings in 3.47s ===============================================
$ pytest tests/models/pvt/test_modeling_pvt.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 111 items / 104 deselected / 7 selected                                                                                          

tests/models/pvt/test_modeling_pvt.py .......                                                                                        [100%]

<warnings redacted>
============================================== 7 passed, 104 deselected, 2 warnings in 9.02s ===============================================
$ pytest tests/models/vit_msn/test_modeling_vit_msn.py -k "offload or parallel"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /root/transformers
configfile: pyproject.toml
plugins: hypothesis-6.98.10, xdist-3.6.1, timeout-2.3.1, anyio-4.3.0
collected 110 items / 103 deselected / 7 selected                                                                                          

tests/models/vit_msn/test_modeling_vit_msn.py .......                                                                                [100%]

<warnings redacted>
============================================== 7 passed, 103 deselected, 2 warnings in 5.37s ===============================================

Who can review?

@amyeroberts

Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

The necessity to move the values onto different devices makes me think that we want to specify more modules in _no_split_modules rather than trying to handle like this.

It might not be possible in all cases, e.g. with the input_ids, but it shouldn't be necessary to do it for the attention modules

@jla524 jla524 marked this pull request as draft April 28, 2024 23:52
@jla524 jla524 marked this pull request as ready for review April 30, 2024 06:57
@jla524
Copy link
Contributor Author

jla524 commented Apr 30, 2024

@amyeroberts Ready for review! All the models are supported using _no_split_modules, and the failed tests in CI seem unrelated (connection timeout).

Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once again, a beautiful and impactful PR.

Thanks for all of your work on this and adding more capabilities to the library's models! ❤️

@amyeroberts amyeroberts merged commit 0ae789e into huggingface:main Apr 30, 2024
19 checks passed
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@jla524 jla524 deleted the more_multidevice branch May 1, 2024 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Community contribution: enabling device_map="auto" support for more vision and multimodal models
3 participants