Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛[BUG]: Race condition in parallel workflows #91

Open
nbren12 opened this issue Oct 31, 2023 · 1 comment
Open

🐛[BUG]: Race condition in parallel workflows #91

nbren12 opened this issue Oct 31, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@nbren12
Copy link
Collaborator

nbren12 commented Oct 31, 2023

Version

main

On which installation method(s) does this occur?

No response

Describe the issue

The model data is downloading by all ranks of a parallel job in parallel. This will result in corrupt files if the files haven't been download into the model registry yet.

Environment details

No response
@nbren12 nbren12 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Oct 31, 2023
@nbren12
Copy link
Collaborator Author

nbren12 commented Dec 19, 2023

@yairchn Just ran into this problem. worth fixing.

@nbren12 nbren12 removed the ? - Needs Triage Need team to review and classify label Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nbren12