Add benchmarks (#549)

Copied the benchmarking setup and one simple benchmark (as an example) from cuml.

Authors:
  - Artem M. Chirkin (https://github.com/achirkin)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Jordan Jacobelli (https://github.com/Ethyling)

URL: #549

BUILD.md

@@ -26,39 +26,52 @@ The recommended way to build and install RAFT is to use the `build.sh` script in
 
 ### <a id="install_header_only_cpp"></a>Header-only C++
 
-RAFT depends on many different core libraries such as `thrust`, `cub`, `cucollections`, and `rmm`, which will be downloaded automatically by `cmake` even when only installing the headers. It's important to note that while all the headers will be installed and available, some parts of the RAFT API depend on libraries like `FAISS`, which can also be downloaded in the RAFT build but will need to be told to do so. 
+RAFT depends on many different core libraries such as `thrust`, `cub`, `cucollections`, and `rmm`, which will be downloaded automatically by `cmake` even when only installing the headers. It's important to note that while all the headers will be installed and available, some parts of the RAFT API depend on libraries like `FAISS`, which can also be downloaded in the RAFT build but will need to be told to do so.
 
 The following example builds and installs raft in header-only mode:
 ```bash
-./build.sh libraft --nogtest
+./build.sh libraft
 ```
 
 ###<a id="shared_cpp_libs"></a>C++ Shared Libraries (optional)
 
-Shared libraries are provided to speed up compile times for larger libraries which may heavily utilize some of the APIs. These shared libraries can also significantly improve re-compile times while developing against the APIs. 
+Shared libraries are provided to speed up compile times for larger libraries which may heavily utilize some of the APIs. These shared libraries can also significantly improve re-compile times while developing against the APIs.
 
 Build all the shared libraries by passing `--compile-libs` flag to `build.sh`:
 
 ```bash
-./build.sh libraft --compile-libs --nogtest
+./build.sh libraft --compile-libs
 ```
- 
+
 To remain flexible, the individual shared libraries have their own flags and multiple can be used (though currently only the `nn` and `distance` packages contain shared libraries):
 ```bash
-./build.sh libraft --compile-nn --compile-dist --nogtest
+./build.sh libraft --compile-nn --compile-dist
 ```
 
 ###<a id="gtests"></a>Googletests
 
-Compile the Googletests by removing the `--nogtest` flag from `build.sh`:
+Compile the Googletests using the `tests` target in `build.sh`:
 ```bash
-./build.sh libraft --compile-nn --compile-dist
+./build.sh libraft tests --compile-nn --compile-dist
 ```
 
 To run C++ tests:
 
 ```bash
-./test_raft
+./cpp/build/test_raft
+```
+
+###<a id="benchmarks"></a>Benchmarks
+
+Compile the benchmarks using the `bench` target in `build.sh`:
+```bash
+./build.sh libraft bench --compile-nn --compile-dist
+```
+
+To run C++ tests:
+
+```bash
+./cpp/build/bench_raft
 ```
 
 ### <a id="cpp_using_cmake"></a>C++ Using Cmake
@@ -77,15 +90,16 @@ RAFT's cmake has the following configurable flags available:.
 
 | Flag | Possible Values | Default Value | Behavior |
 | --- | --- | --- | --- |
-| BUILD_TESTS | ON, OFF | ON | Compile Googletests |  
+| BUILD_TESTS | ON, OFF | ON | Compile Googletests |
+| BUILD_BENCH | ON, OFF | ON | Compile benchmarks |
 | RAFT_COMPILE_LIBRARIES | ON, OFF | OFF | Compiles all `libraft` shared libraries (these are required for Googletests) |
-| RAFT_COMPILE_NN_LIBRARY | ON, OFF | ON | Compiles the `libraft-nn` shared library |  
-| RAFT_COMPILE_DIST_LIBRARY | ON, OFF | ON | Compiles the `libraft-distance` shared library |  
+| RAFT_COMPILE_NN_LIBRARY | ON, OFF | ON | Compiles the `libraft-nn` shared library |
+| RAFT_COMPILE_DIST_LIBRARY | ON, OFF | ON | Compiles the `libraft-distance` shared library |
 | RAFT_ENABLE_NN_DEPENDENCIES | ON, OFF | OFF | Searches for dependencies of nearest neighbors API, such as FAISS, and compiles them if not found. |
-| RAFT_USE_FAISS_STATIC | ON, OFF | OFF | Statically link FAISS into `libraft-nn` | 
+| RAFT_USE_FAISS_STATIC | ON, OFF | OFF | Statically link FAISS into `libraft-nn` |
 | DETECT_CONDA_ENV | ON, OFF | ON | Enable detection of conda environment for dependencies |
 | NVTX | ON, OFF | OFF | Enable NVTX Markers |
-| CUDA_ENABLE_KERNELINFO | ON, OFF | OFF | Enables `kernelinfo` in nvcc. This is useful for `compute-sanitizer` | 
+| CUDA_ENABLE_KERNELINFO | ON, OFF | OFF | Enables `kernelinfo` in nvcc. This is useful for `compute-sanitizer` |
 | CUDA_ENABLE_LINEINFO  | ON, OFF | OFF | Enable the -lineinfo option for nvcc |
 | CUDA_STATIC_RUNTIME | ON, OFF | OFF | Statically link the CUDA runtime |
 
@@ -115,8 +129,8 @@ python setup.py install
 ```
 
 To run the Python tests:
-```bash 
-cd python 
+```bash
+cd python
 python -m pytest raft
 ```
 
@@ -142,14 +156,14 @@ The following example shows how to use the `libraft-distance` API with the pre-c
 
 RAFT uses the [RAPIDS cmake](https://github.com/rapidsai/rapids-cmake) library, so it can be easily included into downstream projects. RAPIDS cmake provides a convenience layer around the [Cmake Package Manager (CPM)](https://github.com/cpm-cmake/CPM.cmake). The following example is similar to building RAFT itself from source but allows it to be done in cmake, providing the `raft::raft` link target and `RAFT_INCLUDE_DIR` for includes. The `COMPILE_LIBRARIES` option enables the building of the shared libraries.
 
-The following `cmake` snippet enables a flexible configuration of RAFT: 
+The following `cmake` snippet enables a flexible configuration of RAFT:
 
 ```cmake
 
 set(RAFT_VERSION "22.04")
 
 function(find_and_configure_raft)
-  set(oneValueArgs VERSION FORK PINNED_TAG USE_FAISS_STATIC 
+  set(oneValueArgs VERSION FORK PINNED_TAG USE_FAISS_STATIC
           COMPILE_LIBRARIES ENABLE_NN_DEPENDENCIES CLONE_ON_PIN
           USE_NN_LIBRARY USE_DISTANCE_LIBRARY)
   cmake_parse_arguments(PKG "${options}" "${oneValueArgs}"
@@ -165,14 +179,14 @@ function(find_and_configure_raft)
   endif()
 
   #-----------------------------------------------------
-  # Add components 
+  # Add components
   #-----------------------------------------------------
 
   string(APPEND RAFT_COMPONENTS "")
   if(PKG_USE_NN_LIBRARY)
     string(APPEND RAFT_COMPONENTS " nn")
   endif()
-  
+
   if(PKG_USE_DISTANCE_LIBRARY)
     string(APPEND RAFT_COMPONENTS " distance")
   endif()
@@ -221,4 +235,4 @@ find_and_configure_raft(VERSION    ${RAFT_VERSION}.00
 
 ### <a id="py_integration"></a>Python/Cython Integration
 
-Once installed, RAFT's Python library can be imported and used directly.
+Once installed, RAFT's Python library can be imported and used directly.

build.sh

@@ -18,25 +18,26 @@ ARGS=$*
 # script, and that this script resides in the repo dir!
 REPODIR=$(cd $(dirname $0); pwd)
 
-VALIDARGS="clean libraft pyraft docs -v -g --noinstall --compile-libs --compile-nn --compile-dist --allgpuarch --nvtx --show_depr_warn -h --nogtest --buildfaiss"
+VALIDARGS="clean libraft pyraft docs tests bench -v -g --noinstall --compile-libs --compile-nn --compile-dist --allgpuarch --nvtx --show_depr_warn -h --buildfaiss"
 HELP="$0 [<target> ...] [<flag> ...]
  where <target> is:
    clean            - remove all existing build artifacts and configuration (start over)
    libraft          - build the raft C++ code only. Also builds the C-wrapper library
                       around the C++ code.
-   pyraft             - build the cuml Python package
+   pyraft           - build the cuml Python package
    docs             - build the documentation
+   tests            - build the tests
+   bench            - build the benchmarks
 
  and <flag> is:
    -v               - verbose build mode
    -g               - build for debug
-   --compile-libs    - compile shared libraries for all components
+   --compile-libs   - compile shared libraries for all components
    --compile-nn     - compile shared library for nn component
    --compile-dist   - compile shared library for distance component
    --allgpuarch     - build for all supported GPU architectures
    --buildfaiss     - build faiss statically into raft
-   --nogtest        - do not build google tests for libraft
-   --noinstall     - do not install cmake targets
+   --noinstall      - do not install cmake targets
    --nvtx           - Enable nvtx for profiling support
    --show_depr_warn - show cmake deprecation warnings
    -h               - print this text
@@ -53,12 +54,13 @@ BUILD_DIRS="${LIBRAFT_BUILD_DIR} ${PY_RAFT_BUILD_DIR} ${PYTHON_DEPS_CLONE}"
 CMAKE_LOG_LEVEL=""
 VERBOSE_FLAG=""
 BUILD_ALL_GPU_ARCH=0
-BUILD_TESTS=YES
+BUILD_TESTS=OFF
+BUILD_BENCH=OFF
 BUILD_STATIC_FAISS=OFF
 COMPILE_LIBRARIES=OFF
 COMPILE_NN_LIBRARY=OFF
 COMPILE_DIST_LIBRARY=OFF
-ENABLE_NN_DEPENDENCIES=${BUILD_TESTS}
+ENABLE_NN_DEPENDENCIES=OFF
 NVTX=OFF
 CLEAN=0
 DISABLE_DEPRECATION_WARNINGS=ON
@@ -110,11 +112,6 @@ fi
 if hasArg --allgpuarch; then
     BUILD_ALL_GPU_ARCH=1
 fi
-if hasArg --nogtest; then
-    BUILD_TESTS=OFF
-    COMPILE_LIBRARIES=OFF
-    ENABLE_NN_DEPENDENCIES=OFF
-fi
 
 if hasArg --compile-libs || (( ${NUMARGS} == 0 )); then
   COMPILE_LIBRARIES=ON
@@ -123,11 +120,24 @@ fi
 if hasArg --compile-nn || hasArg --compile-libs || (( ${NUMARGS} == 0 )); then
     ENABLE_NN_DEPENDENCIES=ON
     COMPILE_NN_LIBRARY=ON
-    CMAKE_TARGET="raft_nn_lib;${CMAKE_TARGET}"
+    CMAKE_TARGET="${CMAKE_TARGET};raft_nn_lib"
 fi
+
 if hasArg --compile-dist || hasArg --compile-libs || (( ${NUMARGS} == 0 )); then
     COMPILE_DIST_LIBRARY=ON
-    CMAKE_TARGET="raft_distance_lib;${CMAKE_TARGET}"
+    CMAKE_TARGET="${CMAKE_TARGET};raft_distance_lib"
+fi
+
+if hasArg tests || (( ${NUMARGS} == 0 )); then
+    BUILD_TESTS=ON
+    ENABLE_NN_DEPENDENCIES=ON
+    CMAKE_TARGET="${CMAKE_TARGET};test_raft"
+fi
+
+if hasArg bench || (( ${NUMARGS} == 0 )); then
+    BUILD_BENCH=ON
+    ENABLE_NN_DEPENDENCIES=ON
+    CMAKE_TARGET="${CMAKE_TARGET};bench_raft"
 fi
 
 if hasArg --buildfaiss; then
@@ -165,7 +175,7 @@ fi
 
 ################################################################################
 # Configure for building all C++ targets
-if (( ${NUMARGS} == 0 )) || hasArg libraft || hasArg docs; then
+if (( ${NUMARGS} == 0 )) || hasArg libraft || hasArg docs || hasArg tests || hasArg bench; then
     if (( ${BUILD_ALL_GPU_ARCH} == 0 )); then
         RAFT_CMAKE_CUDA_ARCHITECTURES="NATIVE"
         echo "Building for the architecture of the GPU in the system..."
@@ -184,17 +194,13 @@ if (( ${NUMARGS} == 0 )) || hasArg libraft || hasArg docs; then
           -DNVTX=${NVTX} \
           -DDISABLE_DEPRECATION_WARNINGS=${DISABLE_DEPRECATION_WARNINGS} \
           -DBUILD_TESTS=${BUILD_TESTS} \
+          -DBUILD_BENCH=${BUILD_BENCH} \
           -DCMAKE_MESSAGE_LOG_LEVEL=${CMAKE_LOG_LEVEL} \
           -DRAFT_COMPILE_NN_LIBRARY=${COMPILE_NN_LIBRARY} \
           -DRAFT_COMPILE_DIST_LIBRARY=${COMPILE_DIST_LIBRARY} \
           -DRAFT_USE_FAISS_STATIC=${BUILD_STATIC_FAISS}
 
-  if (( ${NUMARGS} == 0 )) || hasArg libraft; then
-      # Run all c++ targets at once
-      if ! hasArg --nogtest; then
-        CMAKE_TARGET="${CMAKE_TARGET};test_raft;"
-      fi
-
+  if [[ ${CMAKE_TARGET} != "" ]] || [[ ${INSTALL_TARGET} != "" ]]; then
       echo "-- Compiling targets: ${CMAKE_TARGET}, verbose=${VERBOSE_FLAG}"
       cmake --build  "${LIBRAFT_BUILD_DIR}" ${VERBOSE_FLAG} -j${PARALLEL_LEVEL} --target ${CMAKE_TARGET} ${INSTALL_TARGET}
   fi

ci/checks/style.sh

@@ -43,6 +43,7 @@ fi
 
 # Check for a consistent #include syntax
 HASH_INCLUDE=`python cpp/scripts/include_checker.py \
+                     cpp/bench \
                      cpp/include \
                      cpp/test \
                      2>&1`

ci/gpu/build.sh

@@ -96,9 +96,9 @@ export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
 gpuci_logger "Build C++ and Python targets"
 # These should link against the existing shared libs
 if hasArg --skip-tests; then
-  "$WORKSPACE/build.sh" pyraft libraft -v --nogtest
-else
   "$WORKSPACE/build.sh" pyraft libraft -v
+else
+  "$WORKSPACE/build.sh" pyraft libraft tests bench -v
 fi
 
 gpuci_logger "sccache stats"

conda/recipes/libraft_distance/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
+# Copyright (c) 2022, NVIDIA CORPORATION.
 
-./build.sh libraft -v --allgpuarch --compile-dist --nogtest
+./build.sh libraft -v --allgpuarch --compile-dist

conda/recipes/libraft_headers/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
+# Copyright (c) 2022, NVIDIA CORPORATION.
 
-./build.sh libraft -v --allgpuarch --nogtest
+./build.sh libraft -v --allgpuarch

conda/recipes/libraft_nn/build.sh

@@ -1,3 +1,4 @@
 #!/usr/bin/env bash
+# Copyright (c) 2022, NVIDIA CORPORATION.
 
-./build.sh libraft -v --allgpuarch --compile-nn --nogtest
+./build.sh libraft -v --allgpuarch --compile-nn

cpp/CMakeLists.txt

@@ -41,6 +41,7 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 # - User Options  ------------------------------------------------------------
 
 option(BUILD_TESTS "Build raft unit-tests" ON)
+option(BUILD_BENCH "Build raft C++ benchmark tests" ON)
 option(CUDA_ENABLE_KERNELINFO "Enable kernel resource usage info" OFF)
 option(CUDA_ENABLE_LINEINFO "Enable the -lineinfo option for nvcc (useful for cuda-memcheck / profiler)" OFF)
 option(CUDA_STATIC_RUNTIME "Statically link the CUDA runtime" OFF)
@@ -58,6 +59,7 @@ include(CMakeDependentOption)
 cmake_dependent_option(RAFT_USE_FAISS_STATIC "Build and statically link the FAISS library for nearest neighbors search on GPU" ON RAFT_COMPILE_LIBRARIES OFF)
 
 message(VERBOSE "RAFT: Build RAFT unit-tests: ${BUILD_TESTS}")
+message(VERBOSE "RAFT: Building raft C++ benchmarks: ${BUILD_BENCH}")
 message(VERBOSE "RAFT: Enable detection of conda environment for dependencies: ${DETECT_CONDA_ENV}")
 message(VERBOSE "RAFT: Disable depreaction warnings " ${DISABLE_DEPRECATION_WARNINGS})
 message(VERBOSE "RAFT: Disable OpenMP: ${DISABLE_OPENMP}")
@@ -122,6 +124,10 @@ if(BUILD_TESTS)
   include(cmake/thirdparty/get_ucx.cmake)
 endif()
 
+if(BUILD_BENCH)
+  include(cmake/thirdparty/get_gbench.cmake)
+endif()
+
 ##############################################################################
 # - raft ---------------------------------------------------------------------
 
@@ -411,6 +417,13 @@ if(BUILD_TESTS)
   include(test/CMakeLists.txt)
 endif()
 
+##############################################################################
+# - build benchmark executable -----------------------------------------------
+
+if(BUILD_BENCH)
+  include(bench/CMakeLists.txt)
+endif()
+
 ##############################################################################
 # - doxygen targets ----------------------------------------------------------
 

cpp/bench/CMakeLists.txt

@@ -0,0 +1,60 @@
+#=============================================================================
+# Copyright (c) 2022, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#=============================================================================
+
+set(RAFT_CPP_BENCH_TARGET "bench_raft")
+
+# (please keep the filenames in alphabetical order)
+add_executable(${RAFT_CPP_BENCH_TARGET}
+  bench/linalg/reduce.cu
+  bench/main.cpp
+)
+
+set_target_properties(${RAFT_CPP_BENCH_TARGET}
+  PROPERTIES BUILD_RPATH "\$ORIGIN"
+          # set target compile options
+          CXX_STANDARD                        17
+          CXX_STANDARD_REQUIRED               ON
+          CUDA_STANDARD                       17
+          CUDA_STANDARD_REQUIRED              ON
+          POSITION_INDEPENDENT_CODE           ON
+          INTERFACE_POSITION_INDEPENDENT_CODE ON
+          INSTALL_RPATH "\$ORIGIN/../../../lib"
+)
+
+target_compile_options(${RAFT_CPP_BENCH_TARGET}
+  PRIVATE "$<$<COMPILE_LANGUAGE:CXX>:${RAFT_CXX_FLAGS}>"
+          "$<$<COMPILE_LANGUAGE:CUDA>:${RAFT_CUDA_FLAGS}>"
+)
+
+target_include_directories(${RAFT_CPP_BENCH_TARGET}
+  PUBLIC  "$<BUILD_INTERFACE:${RAFT_SOURCE_DIR}/bench>"
+)
+
+target_link_libraries(${RAFT_CPP_BENCH_TARGET}
+  PRIVATE
+    raft::raft
+    faiss::faiss
+    benchmark::benchmark
+    $<TARGET_NAME_IF_EXISTS:OpenMP::OpenMP_CXX>
+    $<TARGET_NAME_IF_EXISTS:conda_env>
+)
+
+install(
+  TARGETS ${RAFT_CPP_BENCH_TARGET}
+  COMPONENT testing
+  DESTINATION bin/libraft/gbench
+  EXCLUDE_FROM_ALL
+)