Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Check if Dask has quit to avoid throwing an exception and triggering a segfault on ddp exit #3961

Merged

Conversation

alexbarghi-nv
Copy link
Member

Currently, when training with ddp, if dask exits before the CuGraphStore is cleaned up, an exception is thrown, which causes ddp to quit with an error, which then causes a segfault, making users think that the workflow has failed when it has actually succeeded. This bug gracefully displays a warning if the dask dataset can't be deleted, which resolves this issue.

@alexbarghi-nv alexbarghi-nv requested a review from a team as a code owner October 30, 2023 18:14
@alexbarghi-nv alexbarghi-nv self-assigned this Oct 30, 2023
@alexbarghi-nv alexbarghi-nv added the bug Something isn't working label Oct 30, 2023
Copy link
Member

@VibhuJawa VibhuJawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexbarghi-nv alexbarghi-nv added the non-breaking Non-breaking change label Oct 30, 2023
@alexbarghi-nv alexbarghi-nv added this to the 23.12 milestone Oct 30, 2023
Copy link
Member

@tingyu66 tingyu66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alexbarghi-nv
Copy link
Member Author

/merge

@rapids-bot rapids-bot bot merged commit 5c0bc8a into rapidsai:branch-23.12 Nov 1, 2023
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants