[REVIEW] Reduce code duplication for `dask` & `distributed` nightly/stable installs #11565

galipremsagar · 2022-08-18T17:30:38Z

Description

After dask/dask#9367 was fixed in dask upstream we had to bump the minimum version of dask to 2022.8.0 to correctly fetch nightly(if channel exists) or stable (if dask/dev label doesn't exist). Without this fix, conda builds were always picking up 2022.7.1 only and/or there would be a mix of nightly & stable packages in an env.

This PR also does some cleanup and makes the build.sh script easy to maintain.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

ci/gpu/build.sh

Dask-cudf groupby tests *should* be failing as a result of dask/dask#9302 (see [failures](https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cudf/job/prb/job/cudf-gpu-test/CUDA=11.5,GPU_LABEL=driver-495,LINUX_VER=ubuntu20.04,PYTHON=3.9/9946/) in #11565 is merged - where dask/main is being installed correctly). This PR updates the dask_cudf groupby code to fix these failures. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #11561

codecov · 2022-08-19T12:17:03Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.10@e64c2da). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head ff21bb4 differs from pull request most recent head 40ed5e5. Consider uploading reports for the commit 40ed5e5 to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-22.10   #11565   +/-   ##
===============================================
  Coverage                ?   87.51%           
===============================================
  Files                   ?      133           
  Lines                   ?    21798           
  Branches                ?        0           
===============================================
  Hits                    ?    19077           
  Misses                  ?     2721           
  Partials                ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

rjzamora · 2022-08-19T13:17:11Z

Thanks for taking care of this @galipremsagar !

ci/gpu/build.sh

ci/benchmark/build.sh

charlesbluca · 2022-08-23T14:12:47Z

ci/benchmark/build.sh

@@ -39,6 +39,9 @@ export LIBCUDF_KERNEL_CACHE_PATH="$HOME/.jitify-cache"
 # Dask & Distributed option to install main(nightly) or `conda-forge` packages.
 export INSTALL_DASK_MAIN=1

+# Dask version to install when `INSTALL_DASK_MAIN=0`
+export DASK_STABLE_VERSION="2022.8.0"


Want to note that the relevant fixes for the mixed stable/nightly issue here are in conda-forge/dask-feedstock#191 and conda-forge/distributed-feedstock#218, since our problem is that when installing conda-forge dask we ended up pulling in nightly dask-core/distributed.

With this context, conda-forge/conda-forge-repodata-patches-feedstock#312 applies this change across all stable dask/distributed packages that would've had this issue, so once that's in we shouldn't have to bump the stable version here

Awesome, I'll hold off until conda-forge/conda-forge-repodata-patches-feedstock#312 is merged and will revert back to the prev stable version that we were pointing to as a minimum.

Co-authored-by: Charles Blackmon-Luca <[email protected]>

charlesbluca

A couple suggestions so we can verify that the new stable install command works:

ci/benchmark/build.sh

ci/gpu/build.sh

ci/benchmark/build.sh

ci/gpu/build.sh

conda/environments/cudf_dev_cuda11.5.yml

conda/recipes/custreamz/meta.yaml

conda/recipes/dask-cudf/meta.yaml

python/dask_cudf/setup.py

charlesbluca · 2022-08-24T14:16:02Z

After doing some more local testing of conda-forge/conda-forge-repodata-patches-feedstock#312 I realize there's still issues that need to be resolved 🤦🏽

mamba install -c dask/label/dev -c conda-forge dask==2022.7.1
...
  + dask                               2022.7.1  pyhd8ed1ab_0        conda-forge/noarch           5kB
  + dask-core                   2022.7.2a220805  py_g386b4753f_28    dask/label/dev/noarch      864kB
  + distributed                 2022.7.2a220805  py_ge1f3779a_27     dask/label/dev/noarch      758kB

github-actions · 2022-09-23T17:11:46Z

This PR has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this PR if it is no longer required. Otherwise, please respond with a comment indicating any updates. This PR will be labeled inactive-90d if there is no activity in the next 60 days.

Co-authored-by: Charles Blackmon-Luca <[email protected]>

galipremsagar · 2022-09-27T01:47:59Z

@charlesbluca could you review this PR? I know this is not solving the original problem we were trying to address and it has to be done at dask side, but this does remove a lot of version number duplication and will be helpful while we pin & unpin for release.

ci/benchmark/build.sh

ci/gpu/build.sh

Co-authored-by: Charles Blackmon-Luca <[email protected]>

galipremsagar · 2022-09-27T19:23:27Z

@gpucibot merge

temp commit

f620679

galipremsagar added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Aug 18, 2022

github-actions bot added the gpuCI label Aug 18, 2022

Update build.sh

43b944d

galipremsagar changed the title ~~test ci~~ [REVIEW] Fix CI script to install dask & distributed nightly/stable based on flag Aug 18, 2022

galipremsagar changed the title ~~[REVIEW] Fix CI script to install dask & distributed nightly/stable based on flag~~ [REVIEW] Fix CI script to install dask & distributed nightly/stable based on INSTALL_DASK_MAIN Aug 18, 2022

galipremsagar self-assigned this Aug 18, 2022

galipremsagar added 3 - Ready for Review Ready for review by team bug Something isn't working non-breaking Non-breaking change and removed 5 - DO NOT MERGE Hold off on merging; see PR for details labels Aug 18, 2022

galipremsagar marked this pull request as ready for review August 18, 2022 22:23

galipremsagar requested a review from a team as a code owner August 18, 2022 22:23

galipremsagar commented Aug 18, 2022

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

Update ci/gpu/build.sh

8152ab5

rjzamora mentioned this pull request Aug 19, 2022

Fix groupby failures in dask_cudf CI #11561

Merged

3 tasks

Merge branch 'rapidsai:branch-22.10' into test_dask

945fdf5

galipremsagar added 3 commits August 19, 2022 08:58

Update build.sh

31bfa6c

Update build.sh

d65f786

Update meta.yaml

80f8085

galipremsagar added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Aug 19, 2022

github-actions bot added the conda label Aug 19, 2022

galipremsagar commented Aug 22, 2022

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

galipremsagar added 2 commits August 22, 2022 10:53

Update ci/gpu/build.sh

70b0564

fix dask pinnings

ab59187

galipremsagar removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Aug 22, 2022

Merge branch 'rapidsai:branch-22.10' into test_dask

e15e9c5

galipremsagar requested a review from a team as a code owner August 22, 2022 21:00

ajschmidt8 approved these changes Aug 22, 2022

View reviewed changes

charlesbluca reviewed Aug 23, 2022

View reviewed changes

Update ci/benchmark/build.sh

1e19549

Co-authored-by: Charles Blackmon-Luca <[email protected]>

charlesbluca reviewed Aug 24, 2022

View reviewed changes

ci/benchmark/build.sh Show resolved Hide resolved

ci/gpu/build.sh Show resolved Hide resolved

ci/gpu/build.sh Outdated Show resolved Hide resolved

galipremsagar commented Aug 24, 2022

View reviewed changes

Apply suggestions from code review

87ce71c

github-actions bot removed Python Affects Python cuDF API. conda labels Aug 24, 2022

github-actions bot added the inactive-30d label Sep 23, 2022

galipremsagar and others added 3 commits September 26, 2022 15:56

Merge branch 'rapidsai:branch-22.10' into test_dask

b6516df

Update ci/gpu/build.sh

bdc3fed

Co-authored-by: Charles Blackmon-Luca <[email protected]>

Merge branch 'rapidsai:branch-22.10' into test_dask

2c7c957

jakirkham removed the inactive-30d label Sep 26, 2022

galipremsagar changed the title ~~[REVIEW] Fix CI script to install dask & distributed nightly/stable correctly~~ [REVIEW] Reduce code duplication for dask & distributed nightly/stable installs Sep 27, 2022

charlesbluca reviewed Sep 27, 2022

View reviewed changes

ci/benchmark/build.sh Outdated Show resolved Hide resolved

ci/gpu/build.sh Outdated Show resolved Hide resolved

galipremsagar and others added 2 commits September 27, 2022 08:30

Update ci/benchmark/build.sh

6335c68

Co-authored-by: Charles Blackmon-Luca <[email protected]>

Update ci/gpu/build.sh

40ed5e5

Co-authored-by: Charles Blackmon-Luca <[email protected]>

charlesbluca approved these changes Sep 27, 2022

View reviewed changes

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Sep 27, 2022

rapids-bot bot merged commit d8feede into rapidsai:branch-22.10 Sep 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REVIEW] Reduce code duplication for `dask` & `distributed` nightly/stable installs #11565

[REVIEW] Reduce code duplication for `dask` & `distributed` nightly/stable installs #11565

galipremsagar commented Aug 18, 2022 •

edited

Loading

codecov bot commented Aug 19, 2022 •

edited

Loading

rjzamora commented Aug 19, 2022

charlesbluca Aug 23, 2022

galipremsagar Aug 23, 2022

charlesbluca left a comment

charlesbluca commented Aug 24, 2022

github-actions bot commented Sep 23, 2022

galipremsagar commented Sep 27, 2022

galipremsagar commented Sep 27, 2022

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

Conversation

galipremsagar commented Aug 18, 2022 • edited Loading

Description

Checklist

codecov bot commented Aug 19, 2022 • edited Loading

Codecov Report

rjzamora commented Aug 19, 2022

charlesbluca Aug 23, 2022

Choose a reason for hiding this comment

galipremsagar Aug 23, 2022

Choose a reason for hiding this comment

charlesbluca left a comment

Choose a reason for hiding this comment

charlesbluca commented Aug 24, 2022

github-actions bot commented Sep 23, 2022

galipremsagar commented Sep 27, 2022

galipremsagar commented Sep 27, 2022

[REVIEW] Reduce code duplication for `dask` & `distributed` nightly/stable installs #11565

[REVIEW] Reduce code duplication for `dask` & `distributed` nightly/stable installs #11565

galipremsagar commented Aug 18, 2022 •

edited

Loading

codecov bot commented Aug 19, 2022 •

edited

Loading