fix: Add 15 min timeout for downloading cached HF models #3179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vblagoje
requested review from
masci,
sjrl,
a team and
bogdankostic
and removed request for
a team
|
Looks good @vblagoje! My only question would be to ask if we know that when caching works properly if 15 min is typically enough time to download all of the models? It would suck to timeout just because we are downloading a lot of data from HF. |
Yeah, it's more than enough @sjrl. Usually takes a minute or two. I am being really generous here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
There is an ongoing effort to stabilize CI model HF caching. We added
SEGMENT_DOWNLOAD_TIMEOUT_MINenv variable and set it to 15 min. If there are download failures, we will get an exception/timeout rather than wait an hour for the task to timeout. There is an ongoing effort to fix this issue on the GitHub side, and we expect a more permanent fix sometime in September. For more details, see actions/cache#810How did you test it?
No tests have been added.
Notes for the reviewer
See the relevant and referenced issues