-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could we increase disk space of runners? #47
Comments
Thank you! I think we might be able to bump the disk a bit in the 2x and 4x runners. Do you think 80 and 100GB respectively should suffice? cc @aktech The |
I'm not sure TBH, but +20 and 40+ GB sounds reasonable since only one CI job (linux64 + mkl + cuda 11.8) failed while others are still fine! cc: @hmaarrfk |
The question is, what do the build logs actually give us. @jeongseok-meta built the last problematic build and these were the results of the log:
So the RAM usage is quite high, but the disk usage isnt..... this is now puzzling me even more.... |
I'm not sure how much meaningful data can be obtained from the log for a succeeded build, as it may only show disk and RAM usage for a single job (although I'm not certain). However, the log of a failed build actually states that it was unable to write to the disk ("No space left on device").
|
Could we go with a 20+/40+ GB increase, as suggested by @jaimergp? Thank you! |
We are currently facing an issue in OpenStack and I can't apply the config change. @aktech will look into it once he's back from PTO. |
I have bumped the disk for |
Awesome! Thank you for the updating the runners! Sorry for a n00b question, but where could I check the increased disk? @hmaarrfk, let's verify if all pytorch versions (including CUDA 11.8) can build on the new runners without any free disk space issues (in the next build). |
I think you now have the power to do it all yourself! |
I wonder if we could increase the disk space of runners for building large packages, such as pytorch (conda-forge/pytorch-cpu-feedstock#277). One job failed due to insufficient disk space (conda-forge/pytorch-cpu-feedstock#277 (comment)) on the
cirun-openstack-gpu-2xlarge instance
.I have enabled the
free_disk_space
option, but it has caused other build errors due to missing packages (conda-forge/pytorch-cpu-feedstock#277 (comment)). It is unclear whether the conda-forge build scripts need modifications to handle this case though.cc: @Tobias-Fischer, @hmaarrfk
The text was updated successfully, but these errors were encountered: