Adds swap space for ALI runners and report to ci and metrics if it is used #6058
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
facebook-github-bot
added
the
CLA Signed
huydhn
reviewed
Dec 13, 2024
| @@ -138,6 +138,11 @@ fi | |||
|
|
|||
| ${post_install} | |||
|
|
|||
| sudo fallocate -l 3G /swapfile | |||
There was a problem hiding this comment.
Should we have a swap file that scales with the amount of memory on the runner, i.e. https://github.com/pytorch/pytorch/pull/142293/files#diff-b317d4da565a9e329ccf67e669c2ff1f4d4bc5fb0ffa4d74132545ad66f84339R229?
huydhn
approved these changes
Dec 13, 2024
This was referenced Dec 16, 2024
huydhn
added a commit
that referenced
this pull request
Dec 16, 2024
Fix a potential bug from #6058, here is what I see in the CI when it happens https://github.com/pytorch/pytorch/actions/runs/12360769784/job/34497379880?pr=143316#step:5:165
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Dec 17, 2024
A swapfile on Linux runner has been prepared by pytorch/test-infra#6058. So this PR does 2 things: * Start using the swapfile on all Linux build and test jobs * Testing the rollout https://github.com/pytorch-labs/pytorch-gha-infra/pull/582 ### Testing Run `swapon` inside the container and the swapfile shows up correctly: ``` jenkins@259dfb0a314c:~/workspace$ swapon NAME TYPE SIZE USED PRIO /swapfile file 3G 256K -2 ``` Pull Request resolved: #143316 Approved by: https://github.com/ZainRizvi, https://github.com/atalman
aditew01
pushed a commit
to aditew01/pytorch
that referenced
this pull request
Dec 18, 2024
A swapfile on Linux runner has been prepared by pytorch/test-infra#6058. So this PR does 2 things: * Start using the swapfile on all Linux build and test jobs * Testing the rollout https://github.com/pytorch-labs/pytorch-gha-infra/pull/582 ### Testing Run `swapon` inside the container and the swapfile shows up correctly: ``` jenkins@259dfb0a314c:~/workspace$ swapon NAME TYPE SIZE USED PRIO /swapfile file 3G 256K -2 ``` Pull Request resolved: pytorch#143316 Approved by: https://github.com/ZainRizvi, https://github.com/atalman
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a swap space for the autoscaled runners.
Prints on post_job step if the swap usage was detected during the job running, and sends metrics related to swap usage per job.