Separating more namespaces into easier-to-consume sections

README.md

@@ -37,7 +37,7 @@ While not exhaustive, the following general categories help summarize the accele
 All of RAFT's C++ APIs can be accessed header-only and optional pre-compiled shared libraries can 1) speed up compile times and 2) enable the APIs to be used without CUDA-enabled compilers.
 
 In addition to the C++ library, RAFT also provides 2 Python libraries:
-- `pylibraft` - lightweight low-level Python wrappers around RAFT's host-accessible APIs.
+- `pylibraft` - lightweight low-level Python wrappers around RAFT's host-accessible "runtime" APIs.
 - `raft-dask` - multi-node multi-GPU communicator infrastructure for building distributed algorithms on the GPU with Dask.
 
 ## Getting started
@@ -142,7 +142,7 @@ in2 = cp.random.random_sample((n_samples, n_features), dtype=cp.float32)
 output = pairwise_distance(in1, in2, metric="euclidean")
 ```
 
-The `output` array supports [__cuda_array_interface__](https://numba.pydata.org/numba-doc/dev/cuda/cuda_array_interface.html#cuda-array-interface-version-2) so it is interoperable with other libraries like CuPy, Numba, and PyTorch that also support it. 
+The `output` array in the above example is of type `raft.common.device_ndarray`, which supports [__cuda_array_interface__](https://numba.pydata.org/numba-doc/dev/cuda/cuda_array_interface.html#cuda-array-interface-version-2) making it interoperable with other libraries like CuPy, Numba, and PyTorch that also support it. CuPy supports DLPack, which also enables zero-copy conversion from `raft.common.device_ndarray` to JAX and Tensorflow.
 
 Below is an example of converting the output `pylibraft.device_ndarray` to a CuPy array:
 ```python

cpp/include/raft/comms/mpi_comms.hpp

@@ -24,12 +24,27 @@ namespace comms {
 
 using mpi_comms = detail::mpi_comms;
 
+/**
+ * @defgroup mpi_comms_factory MPI Comms Factory Functions
+ * @{
+ */
+
+/**
+ * Given a properly initialized MPI_Comm, construct an instance of RAFT's
+ * MPI Communicator and inject it into the given RAFT handle instance
+ * @param handle raft handle for managing expensive resources
+ * @param comm an initialized MPI communicator
+ */
 inline void initialize_mpi_comms(handle_t* handle, MPI_Comm comm)
 {
   auto communicator = std::make_shared<comms_t>(
     std::unique_ptr<comms_iface>(new mpi_comms(comm, false, handle->get_stream())));
   handle->set_comms(communicator);
 };
 
+/**
+ * @}
+ */
+
 };  // namespace comms
 };  // end namespace raft

cpp/include/raft/comms/std_comms.hpp

@@ -31,8 +31,13 @@ namespace comms {
 using std_comms = detail::std_comms;
 
 /**
- * Function to construct comms_t and inject it on a handle_t. This
- * is used for convenience in the Python layer.
+ * @defgroup std_comms_factory std_comms Factory functions
+ * @{
+ */
+
+/**
+ * Factory function to construct a RAFT NCCL communicator and inject it into a
+ * RAFT handle.
  *
  * @param handle raft::handle_t for injecting the comms
  * @param nccl_comm initialized NCCL communicator to use for collectives
@@ -49,8 +54,8 @@ void build_comms_nccl_only(handle_t* handle, ncclComm_t nccl_comm, int num_ranks
 }
 
 /**
- * Function to construct comms_t and inject it on a handle_t. This
- * is used for convenience in the Python layer.
+ * Factory function to construct a RAFT NCCL+UCX and inject it into a RAFT
+ * handle.
  *
  * @param handle raft::handle_t for injecting the comms
  * @param nccl_comm initialized NCCL communicator to use for collectives
@@ -90,6 +95,10 @@ void build_comms_nccl_ucx(
   handle->set_comms(communicator);
 }
 
+/**
+ * @}
+ */
+
 inline void nccl_unique_id_from_char(ncclUniqueId* id, char* uniqueId, int size)
 {
   memcpy(id->internal, uniqueId, size);

cpp/include/raft/core/comms.hpp

@@ -23,6 +23,11 @@
 namespace raft {
 namespace comms {
 
+/**
+ * @defgroup comms_types Common mnmg comms types
+ * @{
+ */
+
 typedef unsigned int request_t;
 enum class datatype_t { CHAR, UINT8, INT32, UINT32, INT64, UINT64, FLOAT32, FLOAT64 };
 enum class op_t { SUM, PROD, MIN, MAX };
@@ -105,6 +110,15 @@ get_type<double>()
   return datatype_t::FLOAT64;
 }
 
+/**
+ * @}
+ */
+
+/**
+ * @defgroup comms_iface MNMG Communicator Interface
+ * @{
+ */
+
 class comms_iface {
  public:
   virtual ~comms_iface() {}
@@ -215,6 +229,15 @@ class comms_iface {
   virtual void group_end() const = 0;
 };
 
+/**
+ * @}
+ */
+
+/**
+ * @defgroup comms_t Base Communicator Proxy
+ * @{
+ */
+
 class comms_t {
  public:
   comms_t(std::unique_ptr<comms_iface> impl) : impl_(impl.release())
@@ -647,5 +670,9 @@ class comms_t {
   std::unique_ptr<comms_iface> impl_;
 };
 
+/**
+ * @}
+ */
+
 }  // namespace comms
 }  // namespace raft

docs/source/cpp_api.rst

@@ -13,6 +13,7 @@ C++ API
    cpp_api/linalg.rst
    cpp_api/matrix.rst
    cpp_api/mdspan.rst
+   cpp_api/mnmg.rst
    cpp_api/neighbors.rst
    cpp_api/random.rst
    cpp_api/solver.rst

docs/source/cpp_api/cluster.rst

@@ -1,41 +1,17 @@
 Cluster
 =======
 
-This page provides C++ class references for the publicly-exposed elements of the `raft/cluster` headers. RAFT provides
+This page provides C++ API references for the publicly-exposed elements of the `raft/cluster` headers. RAFT provides
 fundamental clustering algorithms which are, themselves, considered reusable building blocks for other algorithms.
 
 .. role:: py(code)
    :language: c++
    :class: highlight
 
-K-Means
-#######
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
 
-``#include <raft/cluster/kmeans.cuh>``
-
-.. doxygennamespace:: raft::cluster::kmeans
-    :project: RAFT
-    :members:
-    :content-only:
-
-
-Hierarchical Clustering
-#######################
-
-``#include <raft/cluster/single_linkage.cuh>``
-
-.. doxygennamespace:: raft::cluster::hierarchy
-    :project: RAFT
-    :members:
-    :content-only:
-
-
-Spectral Clustering
-###################
-
-``#include <raft/spectral/partition.cuh>``
-
-.. doxygennamespace:: raft::spectral
-    :project: RAFT
-    :members:
-    :content-only:
+   cluster_kmeans.rst
+   cluster_slhc.rst
+   cluster_spectral.rst

docs/source/cpp_api/cluster_kmeans.rst

@@ -0,0 +1,16 @@
+K-Means
+=======
+
+This page provides C++ class references for the publicly-exposed elements of the `raft/cluster` headers. RAFT provides
+fundamental clustering algorithms which are, themselves, considered reusable building blocks for other algorithms.
+
+.. role:: py(code)
+   :language: c++
+   :class: highlight
+
+``#include <raft/cluster/kmeans.cuh>``
+
+.. doxygennamespace:: raft::cluster::kmeans
+    :project: RAFT
+    :members:
+    :content-only:

docs/source/cpp_api/cluster_slhc.rst

@@ -0,0 +1,16 @@
+Hierarchical Clustering
+=======================
+
+This page provides C++ class references for the publicly-exposed elements of the `raft/cluster` headers. RAFT provides
+fundamental clustering algorithms which are, themselves, considered reusable building blocks for other algorithms.
+
+.. role:: py(code)
+   :language: c++
+   :class: highlight
+
+``#include <raft/cluster/single_linkage.cuh>``
+
+.. doxygennamespace:: raft::cluster::hierarchy
+    :project: RAFT
+    :members:
+    :content-only:

docs/source/cpp_api/cluster_spectral.rst

@@ -0,0 +1,16 @@
+Spectral Clustering
+===================
+
+This page provides C++ class references for the publicly-exposed elements of the `raft/cluster` headers. RAFT provides
+fundamental clustering algorithms which are, themselves, considered reusable building blocks for other algorithms.
+
+.. role:: py(code)
+   :language: c++
+   :class: highlight
+
+``#include <raft/spectral/partition.cuh>``
+
+.. doxygennamespace:: raft::spectral
+    :project: RAFT
+    :members:
+    :content-only:

docs/source/cpp_api/core.rst

@@ -61,14 +61,3 @@ logger
     :project: RAFT
     :members:
 
-
-Multi-node Multi-GPU
-####################
-
-``#include <raft/core/comms.hpp>``
-
-.. doxygennamespace:: raft::comms
-    :project: RAFT
-    :members:
-    :content-only:
-