Update dask + distributed to 2021.4.0

[jakirkham]

Needed to pick up some serialization bug fixes in the recent Distributed release ( dask/distributed#4645 ) ( dask/distributed#4662 )

[jakirkham]

cc @kkraus14

[raydouglass]

Can you also open a PR against integration to update the versions there as well?

[jakirkham]

These versions appear to be already supported in the integration repo since they are above the minimums. Do we want to bump the minimums?

https://github.com/rapidsai/integration/blob/9c8060e2c20219809591d200de706ef9225b4132/conda/recipes/versions.yaml#L49-L50
https://github.com/rapidsai/integration/blob/9c8060e2c20219809591d200de706ef9225b4132/conda/recipes/versions.yaml#L53-L54

[raydouglass]

Do we want to bump the minimums?

If you're bumping the minimums in cudf, then you should do the same in integration.

[jakirkham]

With distributed, we are bumping the maximum (not minimum). With dask, cuDF has adopted a tighter pinning policy. I'm not sure if we view these the same way as bumping minimums, but wanted to clarify in case something is getting lost here. Happy to do whatever we think is appropriate

[raydouglass]

Happy to do whatever we think is appropriate

Sorry, just to clarify, whatever pinnings we use for dask/distributed in cudf should also be mirrored in integration. Whether its a minimum or maximum change, cudf and integration should mirror the pinnings.

In addition, we should update other repos as well such as cuml

[kkraus14]

In addition, we should update other repos as well such as cuml

cuML shouldn't be forced to follow cuDF's positioning on Dask / Distributed versioning. We need much more fine grained control over versioning in cuDF because dask.dataframe is often being changed alongside cuDF. cuML is generally using more general Dask APIs that don't have nearly the churn where they're safer to pin more loosely.

If someone is only using cuML without dask-cudf this would unnecessarily constrain them.

[codecov]

Codecov Report

Merging #7858 (1de8011) into branch-0.19 (7871e7a) will increase coverage by 0.88%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           branch-0.19    #7858      +/-   ##
===============================================
+ Coverage        81.86%   82.74%   +0.88%     
===============================================
  Files              101      103       +2     
  Lines            16884    17700     +816     
===============================================
+ Hits             13822    14646     +824     
+ Misses            3062     3054       -8     

Impacted Files	Coverage Δ
python/cudf/cudf/utils/dtypes.py	83.44% <0.00%> (-6.08%)	⬇️
python/cudf/cudf/core/column/lists.py	87.32% <0.00%> (-4.08%)	⬇️
python/cudf/cudf/core/column/decimal.py	92.92% <0.00%> (-1.95%)	⬇️
python/cudf/cudf/core/groupby/groupby.py	92.41% <0.00%> (-1.04%)	⬇️
python/cudf/cudf/utils/utils.py	85.36% <0.00%> (-0.07%)	⬇️
python/dask_cudf/dask_cudf/backends.py	89.58% <0.00%> (-0.05%)	⬇️
python/cudf/cudf/__init__.py	100.00% <0.00%> (ø)
python/cudf/cudf/io/feather.py	100.00% <0.00%> (ø)
python/cudf/cudf/core/__init__.py	100.00% <0.00%> (ø)
python/cudf/cudf/utils/ioutils.py	78.71% <0.00%> (ø)
... and 48 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9f1789...1de8011. Read the comment docs.

[raydouglass]

@gpucibot merge