Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] #5003

beckernick · 2022-11-16T21:28:55Z

This PR:

Adds a dependencies.yaml file to auto-generate development conda environment files via the rapids-dependency-file-generator
Adds the generated environment files. I believe these will be helpful in the review, even if we remove them before merging.

beckernick · 2022-11-17T15:41:43Z

I believe this is now ready for initial review. I've marked several dependencies for which I have some open questions with comments in dependencies.yaml.

The questions relate to: 1) build-time or conda package dependencies that are not explicitly included in the current development environment (such as Cython and others); and 2) dependencies for which I'm unsure if we want to include in a generated pip requirements file

beckernick · 2022-11-17T15:47:15Z

This is the diff of the (alphabetically sorted) original 11.5 file and the (sorted) auto-generated file. Note that the bulk of the differences are due to the inclusion of build/runtime dependencies not currently specified in the original yamls but noted in BUILD.md and our runtime conda package requirements. The auto-generated files also change the channel ordering for the CTK 11.2 file to match the others (discussed offline).

sort cuml_dev_cuda11.5.yml > sorted_cuml_dev_cuda11.5.yml
sort all_cuda-115_arch-x86_64.yaml > sorted_all_cuda-115_arch-x86_64.yaml
diff sorted_all_cuda-115_arch-x86_64.yaml sorted_cuml_dev_cuda11.5.yml

1,6c1,11
<   - git+https://github.com/dask/dask-glm@main
<   - git+https://github.com/scikit-learn-contrib/hdbscan.git@master
<   - sparse
< # This file is generated by `rapids-dependency-file-generator`.
< # To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
< - cmake>=3.23.1
---
>
>
>     - git+https://github.com/dask/dask-glm@main
>     - git+https://github.com/scikit-learn-contrib/hdbscan.git@master
>     - sparse
> # To install different versions of packages contained in those meta packages,
> # conda remove --force rapids-build-env rapids-notebook-env rapids-doc-env
> # https://docs.rapids.ai/maintainers/depmgmt/
> # it is recommended to remove those meta packages (without removing the actual
> # packages contained in the environment) first with:
> # rapids-build-env, notebook-env and doc-env are defined in
8c13
< - cuda-python>=11.7.1,<12.0
---
> - cuda-python >=11.7.1,<12.0
11,12d15
< - cupy>=7.8.0,<12.0.0a0
< - cython>=0.29,<0.30
21,22d23
< - gcc_linux-64=9.*
< - joblib>=0.11
28d28
< - nccl>=2.9.9
30d29
< - nvcc_linux-64=11.5
35d33
< - python>=3.8,<3.10
48d45
< - sysroot_linux-64==2.17
56c53
< name: all_cuda-115_arch-x86_64
---
> name: cuml_dev
(dependency-generator) nickbecker-mlt:environments nickbecker$

codecov-commenter · 2022-11-17T21:14:27Z

Codecov Report

Base: 79.38% // Head: 79.44% // Increases project coverage by +0.05% 🎉

Coverage data is based on head (3aa92ea) compared to base (dd197b9).
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff                @@
##           branch-22.12    #5003      +/-   ##
================================================
+ Coverage         79.38%   79.44%   +0.05%     
================================================
  Files               184      184              
  Lines             11698    11698              
================================================
+ Hits               9286     9293       +7     
+ Misses             2412     2405       -7

Flag	Coverage Δ
dask	`45.93% <ø> (ø)`
non-dask	`68.97% <ø> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/cuml/metrics/pairwise_kernels.py	`79.83% <0.00%> (+5.64%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

dantegd

Small thing to avoid the CMake 3.25 bug

dependencies.yaml

…actions-dependencies

beckernick · 2022-11-18T21:08:05Z

dependencies.yaml

+          - cmake>=3.23.1,!=3.25.0 # build dependency not listed in conda dev environment
+          - cuda-python>=11.7.1,<12.0
+          - scikit-build>=0.13.1
+          - cython>=0.29,<0.30 # build dependency not listed in conda dev environment


Reminder for us to remove these comments once we decide whether to keep/remove these dependencies

beckernick · 2022-11-18T21:43:34Z

Running into build slowdowns locally with the generated files vs. the existing conda files. Will do a bit more investigation of the additional dependencies now made explicit.

beckernick · 2022-11-21T22:15:48Z

Build is not slower in the new dev environment. Original and new conda environments both take about 36 minutes on my workstation. I was thrown by seeing the following when building in the new environment, but it turns out I also see it in the old one:

-- Trying "Ninja" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/nicholasb/NVIDIA/cuml/python/_skbuild/linux-x86_64-3.9/cmake-build
  Command:
    cmake /home/nicholasb/NVIDIA/cuml/python -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/home/nicholasb/NVIDIA/cuml/python/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/bin/python -DPYTHON_VERSION_STRING:STRING=3.9.14 -DPYTHON_INCLUDE_DIR:PATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/include/python3.9 -DPYTHON_LIBRARY:FILEPATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib/libpython3.9.so -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib/python3.9/site-packages/skbuild/resources/cmake -DCMAKE_LIBRARY_PATH=/home/nicholasb/NVIDIA/cuml/cpp/build -DCMAKE_MESSAGE_LOG_LEVEL=WARNING -DFIND_CUML_CPP=ON -DCMAKE_BUILD_TYPE:STRING=Release

-- Configuring done
CMake Warning at _skbuild/linux-x86_64-3.9/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/create_modules.cmake:109 (add_library):
  Cannot generate a safe runtime search path for target common_base because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcudart.so.11.0] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcurand.so.10] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcusolver.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcublas.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcublasLt.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcusparse.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  cuml/common/CMakeLists.txt:29 (rapids_cython_create_modules)

removing unnecessary comments from working session w/ Dante

committing latest changes from working session w/ Dante

cjnolet · 2022-11-29T23:20:46Z

@ajschmidt8 are you okay pushing this 23.02?

ajschmidt8 · 2022-12-01T21:15:54Z

@ajschmidt8 are you okay pushing this 23.02?

Yes. I will update the base branch

BUILD.md

dependencies.yaml

python/README.md

dependencies.yaml

Co-authored-by: Bradley Dice <[email protected]>

…actions-dependencies

dantegd · 2022-12-07T20:49:24Z

@gpucibot merge

This fixes a typo I made in a suggestion on #5003. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #5067

…5003) This PR: - Adds a dependencies.yaml file to auto-generate development conda environment files via the rapids-dependency-file-generator - Adds the generated environment files. I believe these will be helpful in the review, even if we remove them before merging. Authors: - Nick Becker (https://github.com/beckernick) - AJ Schmidt (https://github.com/ajschmidt8) - Dante Gama Dessavre (https://github.com/dantegd) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Dante Gama Dessavre (https://github.com/dantegd)

This fixes a typo I made in a suggestion on rapidsai#5003. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#5067

beckernick added 8 commits November 16, 2022 14:44

use AJ's cudf yaml as a skeleton/starting point

0ea34be

first pass at separated dependencies

7aa7ce7

remove extra buildtime gcc requirements from runtime

bd72dfd

python and joblib

72e1ba8

Merge branch 'branch-22.12' into feature/gh-actions-dependencies

a6a6163

full set of CTKs, dependency updates, and comments for open questions

4289e52

formatting

5d31390

add generated environment files

3aa92ea

github-actions bot added the conda conda issue label Nov 17, 2022

beckernick marked this pull request as ready for review November 17, 2022 15:41

beckernick requested a review from a team as a code owner November 17, 2022 15:41

beckernick self-assigned this Nov 17, 2022

rm unused YAML anchors

53833ba

ajschmidt8 changed the title ~~Add dependencies.yaml for rapids-dependency-file-generator~~ Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] Nov 18, 2022

rm existing conda/environment files

2d134c4

ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Nov 18, 2022

ajschmidt8 added 3 commits November 18, 2022 14:08

add GH Action workflow

cc2251e

update README references

93ee470

update update-version.sh

1a49cc8

ajschmidt8 requested a review from a team as a code owner November 18, 2022 19:36

github-actions bot added Cython / Python Cython or Python issue gpuCI gpuCI issue labels Nov 18, 2022

dantegd requested changes Nov 18, 2022

View reviewed changes

dependencies.yaml Outdated Show resolved Hide resolved

ajschmidt8 added 3 commits November 18, 2022 14:56

mv conda exclusive pkgs to conda only lists

89063ed

mv python range to py_version list as default

8223714

Merge remote-tracking branch 'upstream/branch-22.12' into feature/gh-…

2f27957

…actions-dependencies

beckernick commented Nov 18, 2022

View reviewed changes

ajschmidt8 added 2 commits November 22, 2022 15:56

rm comments

28c504f

removing unnecessary comments from working session w/ Dante

commit latest dependencies.yaml changes

d76c5ef

committing latest changes from working session w/ Dante

ajschmidt8 changed the base branch from branch-22.12 to branch-23.02 December 1, 2022 21:16

ajschmidt8 and others added 2 commits December 1, 2022 16:17

Merge branch-23.02 into feature/gh-actions-dependencies

1e31dfc

UPD Update versions to 23.02

088af05

bdice requested changes Dec 7, 2022

View reviewed changes

ajschmidt8 reviewed Dec 7, 2022

View reviewed changes

dependencies.yaml Outdated Show resolved Hide resolved

ajschmidt8 and others added 2 commits December 7, 2022 15:31

Apply suggestions from code review

7859549

Co-authored-by: Bradley Dice <[email protected]>

Merge remote-tracking branch 'upstream/branch-23.02' into feature/gh-…

e5a9c39

…actions-dependencies

ajschmidt8 approved these changes Dec 7, 2022

View reviewed changes

dantegd approved these changes Dec 7, 2022

View reviewed changes

ajschmidt8 merged commit c9ad5c4 into rapidsai:branch-23.02 Dec 7, 2022

bdice mentioned this pull request Dec 7, 2022

Fix link. #5067

Merged

rapids-bot bot pushed a commit that referenced this pull request Dec 8, 2022

Fix link. (#5067)

b82e24b

This fixes a typo I made in a suggestion on #5003. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #5067

dantegd mentioned this pull request Dec 8, 2022

[REVIEW] Unpin dask and distributed for development #5054

Closed

beckernick deleted the feature/gh-actions-dependencies branch December 22, 2022 02:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] #5003

Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] #5003

beckernick commented Nov 16, 2022 •

edited

Loading

beckernick commented Nov 17, 2022

beckernick commented Nov 17, 2022 •

edited

Loading

codecov-commenter commented Nov 17, 2022

dantegd left a comment

beckernick Nov 18, 2022

beckernick commented Nov 18, 2022 •

edited

Loading

beckernick commented Nov 21, 2022 •

edited

Loading

cjnolet commented Nov 29, 2022

ajschmidt8 commented Dec 1, 2022

dantegd commented Dec 7, 2022

Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] #5003

Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci] #5003

Conversation

beckernick commented Nov 16, 2022 • edited Loading

beckernick commented Nov 17, 2022

beckernick commented Nov 17, 2022 • edited Loading

codecov-commenter commented Nov 17, 2022

Codecov Report

dantegd left a comment

Choose a reason for hiding this comment

beckernick Nov 18, 2022

Choose a reason for hiding this comment

beckernick commented Nov 18, 2022 • edited Loading

beckernick commented Nov 21, 2022 • edited Loading

cjnolet commented Nov 29, 2022

ajschmidt8 commented Dec 1, 2022

dantegd commented Dec 7, 2022

beckernick commented Nov 16, 2022 •

edited

Loading

beckernick commented Nov 17, 2022 •

edited

Loading

beckernick commented Nov 18, 2022 •

edited

Loading

beckernick commented Nov 21, 2022 •

edited

Loading