Skip tied weights disk offload test #2782
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
muellerzr
reviewed
May 15, 2024
tests/test_big_modeling.py
Outdated
Comment on lines
487
to
488
| # This test fails because sometimes data_ptr() of compute2.weight is the same of compute1.weight. | ||
| # I check that the values are not the same but it gives the same address. This does not happen on my local machine. |
There was a problem hiding this comment.
If we have enough coverage that seems okay, however we can also do a @flakey decorator that retries the test some amount of times first (maybe 3?)
There was a problem hiding this comment.
I thought about that but I guess it depends more on the hardware. I launch the test a lot of time when debugging and it was failing each time.
There was a problem hiding this comment.
Okay, we can just skip then :)
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yhna940
pushed a commit
to yhna940/accelerate
that referenced
this pull request
May 16, 2024
* skip * fix * quality * fix comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this do ?
This PR skips a flaky test. Not sure why it happens but the
data_ptr()ofcompute2.weightis sometimes the same ofcompute1.weight.Hence it is using the same value as compute1 through self.tied_params_map and we get an assert error. I checked that the values are not the same but it gives the same address. This does not happen on my local machine and it passes sometimes on the CI. Maybe an issue with the hardware. We should have enough coverage withtest_dispatch_model_tied_weights_memory_with_nested_offload_cpu, so I think it's fine if we skip it.cc @fxmarty