Merge branch 'branch-22.12' into 22.12-matrix-test

README.md

@@ -12,19 +12,19 @@ While not exhaustive, the following general categories help summarize the accele
 | Category | Examples |
 | --- | --- |
 | **Data Formats** | sparse & dense, conversions, data generation |
-| **Dense Linear Algebra** | matrix arithmetic, norms, factorization, least squares, svd & eigenvalue problems |
+| **Dense Operations** | linear algebra, matrix and vector operations, slicing, norms, factorization, least squares, svd & eigenvalue problems |
+| **Sparse Operations** | linear algebra, eigenvalue problems, slicing, symmetrization, components & labeling |
 | **Spatial** | pairwise distances, nearest neighbors, neighborhood graph construction |
-| **Sparse Operations** | linear algebra, eigenvalue problems, slicing, symmetrization, labeling |
 | **Basic Clustering** | spectral clustering, hierarchical clustering, k-means |
 | **Solvers** | combinatorial optimization, iterative solvers |
 | **Statistics** | sampling, moments and summary statistics, metrics |
-| **Distributed Tools** | multi-node multi-gpu infrastructure |
+| **Tools & Utilities** | common utilities for developing CUDA applications, multi-node multi-gpu infrastructure |
 
 RAFT provides a header-only C++ library and pre-compiled shared libraries that can 1) speed up compile times and 2) enable the APIs to be used without CUDA-enabled compilers.
 
-RAFT also provides 2 Python libraries:
-- `pylibraft` - low-level Python wrappers around RAFT algorithms and primitives.
-- `raft-dask` - reusable infrastructure for building analytics, including tools for building both single-GPU and multi-node multi-GPU algorithms.
+In addition to the C++ library, RAFT also provides 2 Python libraries:
+- `pylibraft` - lightweight low-level Python wrappers around RAFT algorithms and primitives.
+- `raft-dask` - multi-node multi-GPU communicator infrastructure for building distributed algorithms on the GPU with Dask.
 
 ## Getting started
 
@@ -78,9 +78,9 @@ raft::distance::pairwise_distance(handle, input.view(), input.view(), output.vie
 
 ### Python Example
 
-The `pylibraft` package contains a Python API for RAFT algorithms and primitives. The package is currently limited to pairwise distances, and we will continue adding more.
+The `pylibraft` package contains a Python API for RAFT algorithms and primitives. `pylibraft` integrates nicely into other libraries by being very lightweight with minimal dependencies and accepting any object that supports the `__cuda_array_interface__`, such as [CuPy's ndarray](https://docs.cupy.dev/en/stable/user_guide/interoperability.html#rmm). The package is currently limited to pairwise distances and RMAT graph generation, but we will continue adding more in future releases.
 
-The example below demonstrates computing the pairwise Euclidean distances between cupy arrays. `pylibraft` is a low-level API that prioritizes efficiency and simplicity over being pythonic, which is shown here by pre-allocating the output memory before invoking the `pairwise_distance` function.
+The example below demonstrates computing the pairwise Euclidean distances between CuPy arrays. `pylibraft` is a low-level API that prioritizes efficiency and simplicity over being pythonic, which is shown here by pre-allocating the output memory before invoking the `pairwise_distance` function. Note that CuPy is not a required dependency for `pylibraft`.
 
 ```python
 import cupy as cp
@@ -99,15 +99,15 @@ pairwise_distance(in1, in2, output, metric="euclidean")
 
 ## Installing
 
-RAFT itself can be installed through conda, [Cmake Package Manager (CPM)](https://github.com/cpm-cmake/CPM.cmake), or by building the repository from source. Please refer to the [build instructions](BUILD.md) for more a comprehensive guide on building RAFT and using it in downstream projects.
+RAFT itself can be installed through conda, [Cmake Package Manager (CPM)](https://github.com/cpm-cmake/CPM.cmake), or by building the repository from source. Please refer to the [build instructions](docs/source/build.md) for more a comprehensive guide on building RAFT and using it in downstream projects.
 
 ### Conda
 
 The easiest way to install RAFT is through conda and several packages are provided.
 - `libraft-headers` RAFT headers
 - `libraft-nn` (optional) contains shared libraries for the nearest neighbors primitives.
 - `libraft-distance` (optional) contains shared libraries for distance primitives.
-- `pylibraft` (optional) Python wrappers around RAFT algorithms and primitives
+- `pylibraft` (optional) Python wrappers around RAFT algorithms and primitives.
 - `raft-dask` (optional) enables deployment of multi-node multi-GPU algorithms that use RAFT `raft::comms` in Dask clusters.
 
 Use the following command to install all of the RAFT packages with conda (replace `rapidsai` with `rapidsai-nightly` to install more up-to-date but less stable nightly packages). `mamba` is preferred over the `conda` command.
@@ -119,7 +119,7 @@ You can also install the `libraft-*` conda packages individually using the `mamb
 
 After installing RAFT, `find_package(raft COMPONENTS nn distance)` can be used in your CUDA/C++ cmake build to compile and/or link against needed dependencies in your raft target. `COMPONENTS` are optional and will depend on the packages installed.
 
-### CPM
+### Cmake & CPM
 
 RAFT uses the [RAPIDS-CMake](https://github.com/rapidsai/rapids-cmake) library, which makes it simple to include in downstream cmake projects. RAPIDS CMake provides a convenience layer around CPM. 
 
@@ -186,7 +186,7 @@ mamba activate raft_dev_env
 ./build.sh raft-dask pylibraft libraft tests bench --compile-libs
 ```
 
-The [build](BUILD.md) instructions contain more details on building RAFT from source and including it in downstream projects. You can also find a more comprehensive version of the above CPM code snippet the [Building RAFT C++ from source](BUILD.md#build_cxx_source) section of the build instructions.
+The [build](docs/source/build.md) instructions contain more details on building RAFT from source and including it in downstream projects. You can also find a more comprehensive version of the above CPM code snippet the [Building RAFT C++ from source](docs/source/build.md#building-raft-c-from-source-in-cmake) section of the build instructions.
 
 ## Folder Structure and Contents
 
@@ -198,11 +198,29 @@ The folder structure mirrors other RAPIDS repos, with the following folders:
   - `bench`: Benchmarks source code
   - `cmake`: Cmake modules and templates
   - `doxygen`: Doxygen configuration
-  - `include`: The C++ API headers are fully-contained here
+  - `include`: The C++ API headers are fully-contained here (deprecated directories are excluded from the listing below)
+    - `cluster`: Basic clustering primitives and algorithms.
+    - `comms`: A multi-node multi-GPU communications abstraction layer for NCCL+UCX and MPI+NCCL, which can be deployed in Dask clusters using the `raft-dask` Python package.
+    - `core`: Core API headers which require minimal dependencies aside from RMM and Cudatoolkit. These are safe to expose on public APIs and do not require `nvcc` to build. This is the same for any headers in RAFT which have the suffix `*_types.hpp`. 
+    - `distance`: Distance primitives
+    - `linalg`: Dense linear algebra
+    - `matrix`: Dense matrix operations
+    - `neighbors`: Nearest neighbors and knn graph construction
+    - `random`: Random number generation, sampling, and data generation primitives
+    - `solver`: Iterative and combinatorial solvers for optimization and approximation
+    - `sparse`: Sparse matrix operations
+      - `convert`: Sparse conversion functions
+      - `distance`: Sparse distance computations
+      - `linalg`: Sparse linear algebra
+      - `neighbors`: Sparse nearest neighbors and knn graph construction
+      - `op`: Various sparse operations such as slicing and filtering (Note: this will soon be renamed to `sparse/matrix`)
+      - `solver`: Sparse solvers for optimization and approximation
+    - `stats`: Moments, summary statistics, model performance measures
+    - `util`: Various reusable tools and utilities for accelerated algorithm development
   - `scripts`: Helpful scripts for development
   - `src`: Compiled APIs and template specializations for the shared libraries
   - `test`: Googletests source code
-- `docs`: Source code and scripts for building library documentation (doxygen + pydocs)
+- `docs`: Source code and scripts for building library documentation (Uses breath, doxygen, & pydocs)
 - `python`: Source code for Python libraries.
   - `pylibraft`: Python build and source code for pylibraft library
   - `raft-dask`: Python build and source code for raft-dask library

build.sh

@@ -40,8 +40,8 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
                                   the only option to be supported)
    --minimal-deps              - disables dependencies like thrust so they can be overridden.
                                  can be useful for a pure header-only install
-   --limit-tests               - semicolon-separated list of test executables to compile (e.g. SPATIAL_TEST;CLUSTER_TEST)
-   --limit-bench               - semicolon-separated list of benchmark executables to compute (e.g. SPATIAL_BENCH;CLUSTER_BENCH)
+   --limit-tests               - semicolon-separated list of test executables to compile (e.g. NEIGHBORS_TEST;CLUSTER_TEST)
+   --limit-bench               - semicolon-separated list of benchmark executables to compute (e.g. NEIGHBORS_BENCH;CLUSTER_BENCH)
    --allgpuarch                - build for all supported GPU architectures
    --buildfaiss                - build faiss statically into raft
    --install                   - install cmake targets
@@ -72,8 +72,8 @@ COMPILE_NN_LIBRARY=OFF
 COMPILE_DIST_LIBRARY=OFF
 ENABLE_NN_DEPENDENCIES=OFF
 
-TEST_TARGETS="CLUSTER_TEST;CORE_TEST;DISTANCE_TEST;LABEL_TEST;LINALG_TEST;MATRIX_TEST;RANDOM_TEST;SOLVERS_TEST;SPARSE_TEST;SPARSE_DIST_TEST;SPARSE_NN_TEST;SPATIAL_TEST;STATS_TEST;UTILS_TEST"
-BENCH_TARGETS="CLUSTER_BENCH;SPATIAL_BENCH;DISTANCE_BENCH;LINALG_BENCH;SPARSE_BENCH;RANDOM_BENCH"
+TEST_TARGETS="CLUSTER_TEST;CORE_TEST;DISTANCE_TEST;LABEL_TEST;LINALG_TEST;MATRIX_TEST;RANDOM_TEST;SOLVERS_TEST;SPARSE_TEST;SPARSE_DIST_TEST;SPARSE_NEIGHBORS_TEST;NEIGHBORS_TEST;STATS_TEST;UTILS_TEST"
+BENCH_TARGETS="CLUSTER_BENCH;NEIGHBORS_BENCH;DISTANCE_BENCH;LINALG_BENCH;SPARSE_BENCH;RANDOM_BENCH"
 ENABLE_thrust_DEPENDENCY=ON
 
 CACHE_ARGS=""
@@ -227,18 +227,50 @@ fi
 
 if hasArg tests || (( ${NUMARGS} == 0 )); then
     BUILD_TESTS=ON
-    COMPILE_DIST_LIBRARY=ON
-    ENABLE_NN_DEPENDENCIES=ON
-    COMPILE_NN_LIBRARY=ON
     CMAKE_TARGET="${CMAKE_TARGET};${TEST_TARGETS}"
+
+    # Force compile nn library when needed test targets are specified
+    if [[ $CMAKE_TARGET == *"CLUSTER_TEST"* || \
+          $CMAKE_TARGET == *"SPARSE_DIST_TEST"* || \
+          $CMAKE_TARGET == *"SPARSE_NEIGHBORS_TEST"* || \
+          $CMAKE_TARGET == *"NEIGHBORS_TEST"* || \
+          $CMAKE_TARGET == *"STATS_TEST"* ]]; then
+      echo "-- Enabling nearest neighbors lib for gtests"
+      ENABLE_NN_DEPENDENCIES=ON
+      COMPILE_NN_LIBRARY=ON
+    fi
+
+    # Force compile distance library when needed test targets are specified
+    if [[ $CMAKE_TARGET == *"CLUSTER_TEST"* || \
+          $CMAKE_TARGET == *"DISTANCE_TEST"* || \
+          $CMAKE_TARGET == *"SPARSE_DIST_TEST" || \
+          $CMAKE_TARGET == *"SPARSE_NEIGHBORS_TEST"* || \
+          $CMAKE_TARGET == *"NEIGHBORS_TEST" || \
+          $CMAKE_TARGET == *"STATS_TEST"* ]]; then
+      echo "-- Enabling distance lib for gtests"
+      COMPILE_DIST_LIBRARY=ON
+    fi
 fi
 
 if hasArg bench || (( ${NUMARGS} == 0 )); then
     BUILD_BENCH=ON
-    COMPILE_DIST_LIBRARY=ON
-    ENABLE_NN_DEPENDENCIES=ON
-    COMPILE_NN_LIBRARY=ON
     CMAKE_TARGET="${CMAKE_TARGET};${BENCH_TARGETS}"
+
+    # Force compile nn library when needed benchmark targets are specified
+    if [[ $CMAKE_TARGET == *"CLUSTER_BENCH"* || \
+          $CMAKE_TARGET == *"NEIGHBORS_BENCH"*  ]]; then
+      echo "-- Enabling nearest neighbors lib for benchmarks"
+      ENABLE_NN_DEPENDENCIES=ON
+      COMPILE_NN_LIBRARY=ON
+    fi
+
+    # Force compile distance library when needed benchmark targets are specified
+    if [[ $CMAKE_TARGET == *"CLUSTER_BENCH"* || \
+          $CMAKE_TARGET == *"NEIGHBORS_BENCH"* ]]; then
+      echo "-- Enabling distance lib for benchmarks"
+      COMPILE_DIST_LIBRARY=ON
+    fi
+
 fi
 
 if hasArg --buildfaiss; then

ci/gpu/build.sh

@@ -33,6 +33,12 @@ unset GIT_DESCRIBE_TAG
 # ucx-py version
 export UCX_PY_VERSION='0.29.*'
 
+# Whether to install dask nightly or stable packages.
+export INSTALL_DASK_MAIN=1
+
+# Dask version to install when `INSTALL_DASK_MAIN=0`
+export DASK_STABLE_VERSION="2022.9.2"
+
 ################################################################################
 # SETUP - Check environment
 ################################################################################
@@ -80,11 +86,17 @@ if hasArg --skip-tests; then
     exit 0
 fi
 
-# Install the master version of dask, distributed, and dask-ml
-gpuci_logger "Install the master version of dask and distributed"
 set -x
-pip install "git+https://github.com/dask/[email protected]" --upgrade --no-deps
-pip install "git+https://github.com/dask/[email protected]" --upgrade --no-deps
+# Install latest nightly version for dask and distributed depending on `INSTALL_DASK_MAIN`
+if [[ "${INSTALL_DASK_MAIN}" == 1 ]]; then
+  gpuci_logger "Installing dask and distributed from dask nightly channel"
+  gpuci_mamba_retry install -c dask/label/dev \
+    "dask/label/dev::dask" \
+    "dask/label/dev::distributed"
+else
+  gpuci_logger "gpuci_mamba_retry install conda-forge::dask==${DASK_STABLE_VERSION} conda-forge::distributed==${DASK_STABLE_VERSION} conda-forge::dask-core==${DASK_STABLE_VERSION} --force-reinstall"
+  gpuci_mamba_retry install conda-forge::dask==${DASK_STABLE_VERSION} conda-forge::distributed==${DASK_STABLE_VERSION} conda-forge::dask-core==${DASK_STABLE_VERSION} --force-reinstall
+fi
 set +x
 
 gpuci_logger "Check GPU usage"

conda/environments/raft_dev_cuda11.0.yml

@@ -3,6 +3,7 @@ channels:
 - rapidsai
 - nvidia
 - rapidsai-nightly
+- dask/label/dev
 - conda-forge
 dependencies:
 - c-compiler
@@ -13,6 +14,8 @@ dependencies:
 - clang-tools=11.1.0
 - cython>=0.29,<0.30
 - cmake>=3.23.1
+- dask>=2022.9.2
+- distributed>=2022.9.2
 - scikit-build>=0.13.1
 - rapids-build-env=22.12.*
 - rapids-notebook-env=22.12.*
@@ -30,8 +33,6 @@ dependencies:
 - pip:
     - sphinx_markdown_tables
     - breathe
-    - git+https://github.com/dask/[email protected]
-    - git+https://github.com/dask/[email protected]
 
 # rapids-build-env, notebook-env and doc-env are defined in
 # https://docs.rapids.ai/maintainers/depmgmt/

conda/environments/raft_dev_cuda11.2.yml

@@ -3,6 +3,7 @@ channels:
 - rapidsai
 - nvidia
 - rapidsai-nightly
+- dask/label/dev
 - conda-forge
 dependencies:
 - c-compiler
@@ -13,6 +14,8 @@ dependencies:
 - clang-tools=11.1.0
 - cython>=0.29,<0.30
 - cmake>=3.23.1
+- dask>=2022.9.2
+- distributed>=2022.9.2
 - scikit-build>=0.13.1
 - rapids-build-env=22.12.*
 - rapids-notebook-env=22.12.*
@@ -30,8 +33,6 @@ dependencies:
 - pip:
     - sphinx_markdown_tables
     - breathe
-    - git+https://github.com/dask/[email protected]
-    - git+https://github.com/dask/[email protected]
 
 # rapids-build-env, notebook-env and doc-env are defined in
 # https://docs.rapids.ai/maintainers/depmgmt/

conda/environments/raft_dev_cuda11.4.yml

@@ -3,6 +3,7 @@ channels:
 - rapidsai
 - nvidia
 - rapidsai-nightly
+- dask/label/dev
 - conda-forge
 dependencies:
 - c-compiler
@@ -13,6 +14,8 @@ dependencies:
 - clang-tools=11.1.0
 - cython>=0.29,<0.30
 - cmake>=3.23.1
+- dask>=2022.9.2
+- distributed>=2022.9.2
 - scikit-build>=0.13.1
 - rapids-build-env=22.12.*
 - rapids-notebook-env=22.12.*
@@ -30,8 +33,6 @@ dependencies:
 - pip:
     - sphinx_markdown_tables
     - breathe
-    - git+https://github.com/dask/[email protected]
-    - git+https://github.com/dask/[email protected]
 
 # rapids-build-env, notebook-env and doc-env are defined in
 # https://docs.rapids.ai/maintainers/depmgmt/

conda/environments/raft_dev_cuda11.5.yml

@@ -3,6 +3,7 @@ channels:
 - rapidsai
 - nvidia
 - rapidsai-nightly
+- dask/label/dev
 - conda-forge
 dependencies:
 - c-compiler
@@ -14,6 +15,8 @@ dependencies:
 - clang-tools=11.1.0
 - cython>=0.29,<0.30
 - cmake>=3.23.1
+- dask>=2022.9.2
+- distributed>=2022.9.2
 - scikit-build>=0.13.1
 - rapids-build-env=22.12.*
 - rapids-notebook-env=22.12.*
@@ -31,8 +34,6 @@ dependencies:
 - pip:
     - sphinx_markdown_tables
     - breathe
-    - git+https://github.com/dask/[email protected]
-    - git+https://github.com/dask/[email protected]
 
 # rapids-build-env, notebook-env and doc-env are defined in
 # https://docs.rapids.ai/maintainers/depmgmt/

conda/recipes/raft-dask/meta.yaml

@@ -51,8 +51,8 @@ requirements:
     - ucx >={{ ucx_version }}
     - ucx-py {{ ucx_py_version }}
     - ucx-proc=*=gpu
-    - dask==2022.9.2
-    - distributed==2022.9.2
+    - dask>=2022.9.2
+    - distributed>=2022.9.2
     - cuda-python >=11.5,<11.7.1
     - joblib >=0.11
     - {{ pin_compatible('cudatoolkit', max_pin='x', min_pin='x') }}