raft: Build CUDA 12 packages #1388

vyasr · 2023-03-30T23:36:09Z

First attempt at building libraft with CUDA 12

Closes #1278

conda/recipes/libraft/meta.yaml

Co-authored-by: Bradley Dice <[email protected]>

bdice

Thanks @dantegd! This looks fine to me.

conda/recipes/libraft/meta.yaml

vyasr · 2023-06-29T22:53:19Z

.github/workflows/build.yaml

    with:
      build_type: ${{ inputs.build_type || 'branch' }}
      branch: ${{ inputs.branch }}
      date: ${{ inputs.date }}
      sha: ${{ inputs.sha }}
      skip_upload_pkgs: libraft-template
  docs-build:
-    if: github.ref_type == 'branch'
+    if: github.ref_type == 'branch' && github.event_name == 'push'


Why did this change?

that change was before I started working on the PR, I wonder if @ajschmidt8 might be able to help us here

cpp/bench/ann/CMakeLists.txt

vyasr · 2023-06-29T23:05:44Z

dependencies.yaml

+              - cuda-cudart-dev
+              - cuda-profiler-api
+              - libcublas-dev
+              - libcurand-dev
+              - libcusolver-dev
+              - libcusparse-dev


From what I can see of where the cudatoolkit list is used, most of the places are runtime rather than build envs. Should we be including the non -dev packages instead?

And if we do need the dev packages, do we need cuda-nvcc @bdice?

This is intentional. RAFT is a special case within RAPIDS because it is header-only (like RMM) and has nontrivial library dev dependencies (unlike RMM). We must have the dev packages to build RAFT, but we also list these dev packages as run dependencies for libraft-headers (but not libraft-headers-only) because they are required to use the headers. We can discuss the pros and cons of whether header-only libraries should list their dev dependencies in run (this is a choice that can go either way), but I would prefer to be as consistent as possible with past decisions for CUDA 11 and the current conda recipes for this PR, and most importantly avoid holding back merging for packaging discussions if we can defer them (to prevent blocking cuml/cugraph from starting their CUDA 12 work).

Oh of course wasn't thinking about the header-only bit. OK yeah that makes sense. But then practically speaking we are always going to need cuda-nvcc as well. I'm fine not listing it and leaving that to the user since it's the compiler if that's your preference.

vyasr · 2023-06-29T23:06:19Z

I can't approve or request changes on my own PR, so just leaving comments.

jakirkham

Thanks all! 🙏

Had a few comments/questions below

conda/recipes/raft-dask/meta.yaml

conda/recipes/libraft/meta.yaml

conda/recipes/pylibraft/meta.yaml

conda/recipes/libraft/meta.yaml

Co-authored-by: jakirkham <[email protected]>

…nn-bench

jakirkham

Think the couple CI failures are occurring because we are missing this version constraint

conda/recipes/raft-dask/meta.yaml

Co-authored-by: jakirkham <[email protected]>

jakirkham · 2023-07-03T01:32:58Z

conda/recipes/raft-dask/meta.yaml

-    - {{ compiler('cuda') }} {{ cuda_version }}
+    {% if cuda_major == "11" %}
+    - {{ compiler('cuda11') }} ={{ cuda_version }}
+


Suggesting dropping this blank line (missed it in the previous suggestion 🤦)

Suggested change

Submitted as PR ( #1637 )

cjnolet

LGTM. Thanks everyone who helped make this possible!

dantegd · 2023-07-05T18:41:18Z

/merge

jakirkham · 2023-07-05T18:45:34Z

Thanks everyone! 🙏

Follow up on this review comment ( #1388 (comment) ) Authors: - https://github.com/jakirkham Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Ray Douglass (https://github.com/raydouglass) URL: #1637

vyasr added 4 commits March 30, 2023 16:34

Switch to using cuda 12 workflows

dd487ef

Add nvcc flags needed to work with broken nvidia packages

50d1576

Add a cuda11 compiler version and branch out the recipes

5502f75

Force build and test scripts to install rmm from the PR.

d88346e

vyasr self-assigned this Mar 30, 2023

github-actions bot added the ci label Mar 30, 2023

vyasr added 3 - Ready for Review feature request New feature or request non-breaking Non-breaking change and removed ci labels Mar 30, 2023

vyasr added 2 commits March 30, 2023 16:42

Fix dask branch

6abba37

Put back build deleted lines to see if it fixes CI

6005851

github-actions bot added the ci label Mar 30, 2023

vyasr added 6 commits March 30, 2023 16:46

Add missing quotes

925dac1

Temporarily comment out all unnecessary builds

823868c

Bifurcate all cuda deps by CUDA version

b88b5d0

Add cudart

bc0bd39

Make sure to hard pin the cuda versions during build

a0ef11a

Update versions for 12.0.1 patch release of CUDA 12

60968ff

bdice reviewed Mar 31, 2023

View reviewed changes

conda/recipes/libraft/meta.yaml Show resolved Hide resolved

conda/recipes/libraft/meta.yaml Outdated Show resolved Hide resolved

vyasr added 2 commits March 31, 2023 12:56

Pin librmm

f32bf15

Fix libcusolver version

e7d204f

bdice mentioned this pull request Mar 31, 2023

Build CUDA 12 packages of libraft #1353

Closed

2 tasks

Revert to 12.0.0 versions

6e5e321

vyasr changed the title ~~Build libraft with CUDA 12~~ Build CUDA 12 packages of libraft Mar 31, 2023

vyasr added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Mar 31, 2023

Add cuda runtime to build env for nvcc to see

221b877

vyasr mentioned this pull request Mar 31, 2023

cuml: Build CUDA 12 packages rapidsai/cuml#5318

Merged

vyasr added 2 commits March 31, 2023 15:40

Add missing deps

593619e

Force using nvidia cublas

ac329ff

Apply suggestions from code review

0f2549a

Co-authored-by: Bradley Dice <[email protected]>

jakirkham requested a review from bdice June 29, 2023 21:11

bdice approved these changes Jun 29, 2023

View reviewed changes

conda/recipes/libraft/meta.yaml Outdated Show resolved Hide resolved

vyasr commented Jun 29, 2023

View reviewed changes

bdice mentioned this pull request Jun 30, 2023

FAISS CUDA 12 support for benchmarks. #1627

Open

jakirkham reviewed Jun 30, 2023

View reviewed changes

raydouglass approved these changes Jun 30, 2023

View reviewed changes

Apply suggestions from code review

4dfaee3

Co-authored-by: jakirkham <[email protected]>

bdice approved these changes Jun 30, 2023

View reviewed changes

FIX More review comments applied

f3c482a

jakirkham requested a review from cjnolet June 30, 2023 18:32

This comment was marked as resolved.

Sign in to view

dantegd and others added 2 commits June 30, 2023 14:10

Update conda/recipes/raft-dask/meta.yaml

a3ba165

Co-authored-by: jakirkham <[email protected]>

FIX add ctk back to host of pylibraft and raft-dask

f15d380

dantegd removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Jun 30, 2023

This comment was marked as resolved.

Sign in to view

FIX changes from #1

fa1d5b4

jakirkham approved these changes Jun 30, 2023

View reviewed changes

FIX remove unnecessary cuda libraries for libraft-tests and libraft-a…

fa45ec3

…nn-bench

jakirkham reviewed Jun 30, 2023

View reviewed changes

conda/recipes/raft-dask/meta.yaml Outdated Show resolved Hide resolved

Update conda/recipes/raft-dask/meta.yaml

f264b9b

Co-authored-by: jakirkham <[email protected]>

jakirkham reviewed Jul 3, 2023

View reviewed changes

cjnolet approved these changes Jul 3, 2023

View reviewed changes

Merge branch 'branch-23.08' into feat/cuda12_nvidia_build

40417d9

rapids-bot bot merged commit 32b3dba into rapidsai:branch-23.08 Jul 5, 2023

jakirkham mentioned this pull request Jul 5, 2023

Drop blank line in raft-dask/meta.yaml #1637

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raft: Build CUDA 12 packages #1388

raft: Build CUDA 12 packages #1388

vyasr commented Mar 30, 2023 •

edited by cjnolet

Loading

bdice left a comment

vyasr Jun 29, 2023

dantegd Jun 29, 2023

vyasr Jun 29, 2023

bdice Jun 30, 2023 •

edited

Loading

vyasr Jun 30, 2023

vyasr commented Jun 29, 2023

jakirkham left a comment

This comment was marked as resolved.

This comment was marked as resolved.

jakirkham left a comment

jakirkham Jul 3, 2023 •

edited

Loading

jakirkham Jul 5, 2023

cjnolet left a comment

dantegd commented Jul 5, 2023

jakirkham commented Jul 5, 2023

raft: Build CUDA 12 packages #1388

raft: Build CUDA 12 packages #1388

Conversation

vyasr commented Mar 30, 2023 • edited by cjnolet Loading

bdice left a comment

Choose a reason for hiding this comment

vyasr Jun 29, 2023

Choose a reason for hiding this comment

dantegd Jun 29, 2023

Choose a reason for hiding this comment

vyasr Jun 29, 2023

Choose a reason for hiding this comment

bdice Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

vyasr Jun 30, 2023

Choose a reason for hiding this comment

vyasr commented Jun 29, 2023

jakirkham left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

jakirkham left a comment

Choose a reason for hiding this comment

jakirkham Jul 3, 2023 • edited Loading

Choose a reason for hiding this comment

jakirkham Jul 5, 2023

Choose a reason for hiding this comment

cjnolet left a comment

Choose a reason for hiding this comment

dantegd commented Jul 5, 2023

jakirkham commented Jul 5, 2023

vyasr commented Mar 30, 2023 •

edited by cjnolet

Loading

bdice Jun 30, 2023 •

edited

Loading

jakirkham Jul 3, 2023 •

edited

Loading