[REVIEW] Unpin `dask` & `distributed` for development #10623

galipremsagar · 2022-04-08T15:40:47Z

This PR unpins dask & distributed for development.

codecov · 2022-04-08T16:51:35Z

Codecov Report

Merging #10623 (0d716e1) into branch-22.06 (bf4ffc9) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@               Coverage Diff                @@
##           branch-22.06   #10623      +/-   ##
================================================
+ Coverage         86.33%   86.36%   +0.02%     
================================================
  Files               140      140              
  Lines             22289    22289              
================================================
+ Hits              19244    19250       +6     
+ Misses             3045     3039       -6

Impacted Files	Coverage Δ
python/dask_cudf/dask_cudf/backends.py	`86.44% <100.00%> (ø)`
python/dask_cudf/dask_cudf/sorting.py	`93.03% <100.00%> (ø)`
python/cudf/cudf/core/column/string.py	`89.10% <0.00%> (+0.12%)`	⬆️
python/cudf/cudf/core/groupby/groupby.py	`91.72% <0.00%> (+0.22%)`	⬆️
python/cudf/cudf/core/tools/datetimes.py	`84.49% <0.00%> (+0.30%)`	⬆️
python/cudf/cudf/core/column/lists.py	`92.70% <0.00%> (+1.28%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28aa895...0d716e1. Read the comment docs.

charlesbluca · 2022-04-08T18:09:08Z

ci/benchmark/build.sh

+    gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall"
+    gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall


I've been wondering if a less cumbersome way of achieving this would be to manually remove dask/label/dev from our list of channels here? Maybe something like

Suggested change

gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall"

gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall

gpuci_logger "gpuci_mamba_retry install dask>=2022.03.0 --force-reinstall"

gpuc_conda_retry config --remove channels dask/label/dev

gpuci_mamba_retry install dask>=2022.03.0 --force-reinstall

Not sure if removing the channel would persist in other runs though, cc @ajschmidt8 @jakirkham if this solution makes sense

I remember trying something similar previously but didn't seem to work, testing it now again in CI

Yeah agree. Recall Prem tried several things, but don't recall if this one (and if so whether we encountered any issues with that)

If this doesn't work, happy to merge in as is since the current setup is working

Will this actually install Dask main or just the latest release?

We use nightlies from the Dask channel in some cases. Depends if the channel dask/label/dev is used

And do nightlies really mean "once every night", or does it mean it's a new package after every merge?

Every merge. They are equivalent to using main, but also include Dask's dependencies

Sounds good, thanks for confirming.

@charlesbluca It didn't work the way we would expect it to just like last time, log here: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-22-06/job/cudf/job/prb/job/cudf-gpu-test/CUDA=11.0,GPU_LABEL=driver-450,LINUX_VER=centos7,PYTHON=3.8/150/consoleText
hence reverting these temp changes.

Changes to be in-line with: rapidsai/cudf#10623 Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Sevag Hanssian (https://github.com/sevagh) - https://github.com/jakirkham - Peter Andreas Entschev (https://github.com/pentschev) URL: #892

ci/gpu/build.sh

ci/benchmark/build.sh

ci/gpu/build.sh

python/dask_cudf/dask_cudf/backends.py

galipremsagar · 2022-04-11T16:22:27Z

python/dask_cudf/dask_cudf/sorting.py

@@ -294,6 +294,6 @@ def sort_values(
    df4 = df3.map_partitions(sort_function, **sort_kwargs)
    if not isinstance(divisions, gd.DataFrame) and set_divisions:
        # Can't have multi-column divisions elsewhere in dask (yet)
-        df4.divisions = methods.tolist(divisions)
+        df4.divisions = tuple(methods.tolist(divisions))


This fix is required due to this upstream change: dask/dask#8806

charlesbluca

LGTM; one small question that can be addressed outside of this PR:

python/dask_cudf/dask_cudf/backends.py

galipremsagar · 2022-04-11T17:39:44Z

@gpucibot merge

unpin dask for development

05c6828

galipremsagar added 3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Apr 8, 2022

galipremsagar requested a review from a team as a code owner April 8, 2022 15:40

galipremsagar self-assigned this Apr 8, 2022

galipremsagar requested a review from a team as a code owner April 8, 2022 15:40

github-actions bot added conda Python Affects Python cuDF API. labels Apr 8, 2022

This was referenced Apr 8, 2022

Unpin dask & distributed for development rapidsai/integration#465

Merged

[REVIEW] Unpin dask & distributed for development rapidsai/dask-cuda#892

Merged

This was referenced Apr 8, 2022

[REVIEW] Unpin dask & distributed for development rapidsai/raft#616

Merged

[REVIEW] Unpin dask & distributed for development rapidsai/cuml#4693

Merged

[REVIEW] Unpin dask & distributed for devlopment rapidsai/cugraph#2214

Merged

charlesbluca approved these changes Apr 8, 2022

View reviewed changes

sevagh approved these changes Apr 8, 2022

View reviewed changes

jakirkham approved these changes Apr 8, 2022

View reviewed changes

charlesbluca reviewed Apr 8, 2022

View reviewed changes

test

7d43c63

galipremsagar added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Apr 8, 2022

galipremsagar added 3 commits April 8, 2022 14:27

Update build.sh

7ff2d14

Update build.sh

5934077

Update build.sh

c08481a

galipremsagar commented Apr 8, 2022

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

Apply suggestions from code review

a960653

galipremsagar commented Apr 11, 2022

View reviewed changes

ci/benchmark/build.sh Outdated Show resolved Hide resolved

ci/benchmark/build.sh Outdated Show resolved Hide resolved

ci/gpu/build.sh Outdated Show resolved Hide resolved

galipremsagar added 4 commits April 11, 2022 09:56

Apply suggestions from code review

dbce891

Merge remote-tracking branch 'upstream/branch-22.06' into unpin_dask

c644c10

fix divisions assignment

f23d264

fix category dtype metadata creation

0d716e1

galipremsagar commented Apr 11, 2022

View reviewed changes

python/dask_cudf/dask_cudf/backends.py Show resolved Hide resolved

galipremsagar commented Apr 11, 2022

View reviewed changes

galipremsagar removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Apr 11, 2022

charlesbluca approved these changes Apr 11, 2022

View reviewed changes

python/dask_cudf/dask_cudf/backends.py Show resolved Hide resolved

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Apr 11, 2022

rapids-bot bot merged commit 97ac349 into rapidsai:branch-22.06 Apr 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REVIEW] Unpin `dask` & `distributed` for development #10623

[REVIEW] Unpin `dask` & `distributed` for development #10623

galipremsagar commented Apr 8, 2022

codecov bot commented Apr 8, 2022 •

edited

Loading

charlesbluca Apr 8, 2022

galipremsagar Apr 8, 2022

jakirkham Apr 8, 2022

charlesbluca Apr 8, 2022

pentschev Apr 8, 2022

jakirkham Apr 8, 2022

pentschev Apr 8, 2022

jakirkham Apr 8, 2022

pentschev Apr 8, 2022

galipremsagar Apr 11, 2022

galipremsagar Apr 11, 2022

charlesbluca left a comment

galipremsagar commented Apr 11, 2022

		gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall"
		gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall

-    gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall"
-    gpuci_mamba_retry install conda-forge::dask>=2022.03.0 conda-forge::distributed>=2022.03.0 conda-forge::dask-core>=2022.03.0 --force-reinstall
+    gpuci_logger "gpuci_mamba_retry install dask>=2022.03.0 --force-reinstall"
+    gpuc_conda_retry config --remove channels dask/label/dev
+    gpuci_mamba_retry install dask>=2022.03.0 --force-reinstall

[REVIEW] Unpin dask & distributed for development #10623

[REVIEW] Unpin dask & distributed for development #10623

Conversation

galipremsagar commented Apr 8, 2022

codecov bot commented Apr 8, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charlesbluca left a comment

Choose a reason for hiding this comment

galipremsagar commented Apr 11, 2022

[REVIEW] Unpin `dask` & `distributed` for development #10623

[REVIEW] Unpin `dask` & `distributed` for development #10623

codecov bot commented Apr 8, 2022 •

edited

Loading