Fix MG PLC algos intermittent hang #2607
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## branch-22.10 #2607 +/- ##
===============================================
Coverage ? 60.07%
===============================================
Files ? 112
Lines ? 6154
Branches ? 0
===============================================
Hits ? 3697
Misses ? 2457
Partials ? 0 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
| # the same PLC graph, the current iteration might try to cache | ||
| # the past iteration's futures and this can cause a hang if some | ||
| # of those futures get released midway | ||
| del result |
There was a problem hiding this comment.
I think removing result is still a good practice as when cudf_result is completed we will still have result in memory so from a memory relief standpoint it makes sense.
|
|
||
| ddf = dask_cudf.from_delayed(cudf_result).persist() | ||
| wait(ddf) | ||
|
|
There was a problem hiding this comment.
I suggest we also del cudf_result here to save on memory . We have a renumbered call after this so result does not go out of scope immediately, hence its important .
|
rerun tests |
1 similar comment
|
rerun tests |
|
rerun tests |
jnke2016
changed the title
|
rerun tests |
jnke2016
changed the title
|
@gpucibot merge |
Dask doesn't always release some of the inactive futures fast enough. This can be problematic when running the same
algoseveral times with the samePLC graphbecause those futures can be cache in the next iteration causing a hang if some get released midway.This PR manually delete inactive futures.
closes #2568