Skip 2nd serialization pass of DeviceSerialized
#309
Conversation
jakirkham
force-pushed
the
drop_2nd_ser_pass
branch
2 times, most recently
from
e3754eb to
dc4f3c0
Compare
As `"dask"` serialization already converts a CUDA object into headers and frames that Dask is able to work with, drop code that tries to serialize frames on host further (as they are already as simple as they can be). Cuts a fair bit of boilerplate from the spilling path, which should simplify things a bit.
jakirkham
force-pushed
the
drop_2nd_ser_pass
branch
from
dc4f3c0 to
f36593e
Compare
Codecov Report
@@ Coverage Diff @@
## branch-0.15 #309 +/- ##
===============================================
- Coverage 59.29% 59.19% -0.10%
===============================================
Files 16 16
Lines 1275 1272 -3
===============================================
- Hits 756 753 -3
Misses 519 519
Continue to review full report at Codecov.
|
|
So this works now. Though will wait until Distributed 2.18.0 is out so we can bump the dependency. Should be out later today or early next week. Leaving as draft/WIP until then. |
|
Is 2.18.0 strictly necessary for this? |
|
Yeah we need the fix in PR ( dask/distributed#3639 ). This is what we determined when trying this previously ( #256 (comment) ). |
jakirkham
changed the title
DeviceSerializedDeviceSerialized
Ok, makes sense then. Thanks @jakirkham ! |
|
@rapidsai/ops-codeowners could you approve this? |
|
Marked as draft as there are some orthogonal issues that were found with Dask + Distributed 2.18.0. So want to hold off on bumping the requirement/merging until we have addressed those. |
The underlying Distributed issue has been identified and filed as issue ( dask/distributed#3920 ). This is being addressed by PR ( dask/distributed#3922 ). |
|
Marking as ready for review as the orthogonal issues have been fixed upstream. |
|
Good point. Submitted an update in PR ( rapidsai/integration#64 ). |
|
Thank you all! Merging in |
Requires PR ( dask/distributed#3639 ) (so latest Dask release)
Finishes the work in PR ( #268 )
As
"dask"serialization already converts a CUDA object into headers and frames that Dask is able to work with, drop code that tries to serialize frames on host further (as they are already as simple as they can be). Cuts a fair bit of boilerplate from the spilling path, which should simplify things a bit.