Introduce nightly runners #410
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
LysandreJik
reviewed
May 30, 2022
| - name: Update clone | ||
| run: | | ||
| source activate accelerate | ||
| git config --global --add safe.directory '*' |
There was a problem hiding this comment.
I was getting odd permission errors, and this was the documented solution I found. (Will post the link in the AM)
There was a problem hiding this comment.
@LysandreJik here's the specific thread: actions/checkout#760 (comment)
How we got here is the runner has a clone of accelerate inside the working directory for the action, and that clone is then updated, checked out, etc. But we run into this permission denied unsafe repository from github, so github needs to be told we can safely adjust the repository.
* Install git * Fix CPU image as well
sgugger
approved these changes
May 31, 2022
* Fix CUDA tests * Use num_processes to keep everything under one test
* Update test requirements to include psutil, tensorboard, and the right tensorflow version
LysandreJik
approved these changes
May 31, 2022
| - name: Update clone | ||
| run: | | ||
| source activate accelerate | ||
| git config --global --add safe.directory '*' |
muellerzr
added a commit
that referenced
this pull request
May 31, 2022
* Introduce nightly builds * Fixup docker images slightly * Make device-count specific test use `torch.cuda.device_count()` rather than `Accelerator.num_processes` to avoid bug.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce nightly runners
Important
Should be merged after:
What does this add?
This PR adds nightly runners that will test on a GPU and multi-GPU, and runs all slow tests. In total it should take somewhere between 5-10 minutes to run (depending on whether a cache of a Docker image needed to be downloaded beforehand)
Why is it needed?
We currently have no runners for testing CUDA and multi-GPU tests, or slow tests on the CI. As a result there were many undiscovered errors and bugs that needed to be solved. This PR ensures that we can trust the quality of our code and know when we have issues on platforms other than on the CPU.
Currently they are running nightly, a different workflow will handle when pushes to main occur that I will work on later next week.