Simplify distance/detail to make is easier to dispatch to different kernel implementations

[ahendriksen]

The pairwise distance metrics are quite varied. The table below summarizes the differences, in terms of

• Epilog : whether the metric has a non-empty epilog operation.
• Uses norms: whether the metric requires precalculation of the norms of the vectors.
• Has params: whether the norm has additional parameters. The L2 metric, for instance, has the sqrt boolean parameter that determines whether to calculate the squared or actual distance.
• Pre- & post-processing: For some metrics, the norms have to be precalculated. For other metrics, the input matrices are transformed before the kernel launch, and "untransformed" after.
• Expensive inner loop: some metrics use pow, log or other expensive functions in the inner loop.
• Depends on row-major: the calculation of some metrics depend on whether the input is row-major.
• CUTLASS: some metrics have an implementation using CUTLASS and tensor cores.

Metric	Epilog	Uses norms	Has params	Pre- & post-processing	Expensive inner loop	Depends on row-major	CUTLASS
Canberra					x
Chebyshev (Linf)
Correlation	x	x (twice)	x (many)	compute norms		x
Cosine	x	x		compute norms			x
Hamming	x		x (k)
Hellinger	x			sqrt and square
Jensen Shannon	x				x
KL divergence	x		x (row major, x == y)	yes	x	x
L1
L2 expanded	x	x	x (sqrt)	compute norms			x
L2 unexpanded	x		x (sqrt)
Minkowski (Lp)	x		x (p)		x
Russel-Rao	x		x (k, 1/k)

To keep the complexity that results from all these differences in check, there are several layers between the public API and the kernel launch, each with their own responsibility.

Before

1. raft::distance::pairwise_distance takes distance type as a run-time argument and dispatches to raft::distance::detail::pairwise_distance_impl.
2. raft::distance::detail::pairwise_distance_impl allocates workspace as necessary and calls raft::distance::detail::distance
3. raft::distance::detail::distance defines a default final operation (the identity) and calls an overload of itself.
4. raft::distance::detail::distance (with fin_op) initializes a DistanceImpl zero-sized struct with the correct template arguments and runs the .run() method of the struct.
5. raft::distance::detail::DistanceImpl<DistanceType>.run() calls raft::distance::detail::XX_Impl.
6. raft::distance::detail::XX_Impl has the following responsibilities:
   • Pre-compute norms if necessary
   • Transform input if necessary
   • If metric supports a CUTLASS operation, dispatch if necessary.
   • Swap inputs if column-major.
   • Based on runtime parameter row_major dispatch to function template raft::distance::detail::XX<bool row_major>
7. raft::distance::detail::XX based on alignment of input data dispatch to function template raft::distance::detail::XX_Impl<int veclen> (different overload of previous raft::distance::detail::XX_Impl)
8. raft::distance::detail::XX_Impl has the following responsibilities:
   • Define core_op and epilog_op
   • Define use_norms
   • Launch kernel pairwiseDistanceMatKernel with correct launch parameters

Observations:

• Steps 6 and 7 both convert a runtime value to a compile time constant (row-major layout and alignment).
• Step 7 is repeated (copy pasted) for each metric.
• Steps 7 and 8 do a lot of different things and the steps in between do relatively little.
• Steps 1-5 do fairly little (but require a lot of boilerplate)

Proposal:

1. Collect as much of the runtime behavior of each metric in a distance_op that contains:
   • The core_op
   • The epilog_op
   • The required shared memory
   • Whether the inner loop is expensive (and thus loop unrolling should be curtailed)
2. Collect the runtime -> compile-time dispatch in one location (dispatch.cuh)
3. Collect kernel launching in one location
4. Remove some of the boilerplate in steps 1-5.

After

1. raft::distance::pairwise_distance takes distance type as a run-time argument, allocates workspace as necessary, and dispatches to raft::distance::detail::distance.
2. raft::distance::detail::distance defines a default final operation (the identity) and calls an overload of itself.
3. raft::distance::detail::distance (with fin_op) calls an overload of raft::distance::detail::distance_impl for the correct distance type.
4. raft::distance::detail::distance_impl has the following responsibilities:
   • Pre-compute norms if necessary
   • Initialize distance op with parameters as necessary, see below for more information.
   • Transform input if necessary
   • If metric supports a CUTLASS operation, dispatch if necessary.
   • Dispatch to raft::distance::detail::distance_matrix_dispatch
5. raft::distance::detail::distance_matrix_dispatch has the following responsibilities:
   • swap x, y matrices if column major
   • dispatch to correct kernel based on run-time parameters row_major and vec_len
   • Determine kernel policy based on parameters
   • Call raft::distance::detail::pairwise_matrix
6. raft::distance::detail::pairwise_matrix launches the raft::distance::detail::pairwise_matrix_kernel with the correct launch parameters.

Distance_op
raft::distance::detail::ops::XX_distance_op [example] has the following responsibilities:

• Take any parameters (sqrt, k, etc)
• Define core_op and epilog_op
• Define use_norms, expensive_inner_loop, and shared_mem_size().

Still TODO:

• Rename Minkowski and Chebyshev to Lp and Linf.
• Do something with this note in the comments: "if workspace is passed as nullptr, this will return in worksize, the number of bytes of workspace required", which is wrong.
• Add a mechanism to limit duplicate compilation when a CUTLASS kernel is available. This is done in follow up PR Add dispatch based on compute architecture #1295.
• Some distance_ops have additional template parameters. This must be cleared up.

[ahendriksen]

Reminder/TODO

make grid stride loop variables local variables instead of member variables.

See: #838 (comment)

[ahendriksen]

TODO: #838 (comment)

Also look at fused and maskedL2NN for shared memory calculations.

[ahendriksen]

The CI error does not seem related to the changes in the PR:

2023-02-22T19:37:49.5526382Z   Error compiling Cython file:
2023-02-22T19:37:49.5533415Z   ------------------------------------------------------------
2023-02-22T19:37:49.5539691Z   ...
2023-02-22T19:37:49.5546449Z   # cython: language_level = 3
2023-02-22T19:37:49.5552762Z 
2023-02-22T19:37:49.5559417Z 
2023-02-22T19:37:49.5566630Z   from libcpp.memory cimport shared_ptr, unique_ptr
2023-02-22T19:37:49.5573013Z 
2023-02-22T19:37:49.5580739Z   from rmm._lib.cuda_stream_pool cimport cuda_stream_pool

Is there a way to rerun the CI without pushing an empty commit?

[vyasr]

@ahendriksen if you click on the "Details" button it will take you to the actions tab where you can use the "Re-run jobs" dropdown. See https://docs.github.com/en/actions/managing-workflow-runs/re-running-workflows-and-jobs for more info. I've queued up a rerun.

[ahendriksen]

Thanks! And thanks for linking to the docs. I will have a look to see what else I am missing.

[cjnolet]

I really like the idea of abstracting each unique distance formula into composable ops. In this case, not only does it allow us to specify and react differently to various performance conditions, but it allows us to capture the differences of the computations themselves in a single place. This is very much what we did w/ the sparse APIs as well- they are a series of composable operations that can be executed according to their needs (in the sparse case that's binary vs dot-product-based vs full-pairwise evaluation).

I think this looks great. I'd like to see the functions in the public API that accept explicit workspaces deprecated because the memory resource should be getting propagated through the raft::resources instance (and be controllable by the user). I think we have an opportunity to here to expand our developer guide as well. Otherwise, I'm completely on board w/ this change.

[tfeher]

Thanks Allard for this PR! This looks great! I am very happy to see the reduction of code duplication. I have just a few smaller questions.

[tfeher]

Thanks Allard for the updates! The PR looks good to me!

[cjnolet]

Just a heads up- we need to merge this cuml PR before this RAFT PR is merged, otherwise cuml will break downstream.

[ahendriksen]

Good catch. I should be more careful with the non-breaking label. Although that file has been deprecated for a couple of months now.

[cjnolet]

Good catch. I should be more careful with the non-breaking label. Although that file has been deprecated for a couple of months now.

No problem. To be fair, we had updated cuml in 23.02 to remove most of its uses of deprecated headers but this one slipped through the cracks. Before we merged this I just wanted to do a quick grep to be sure.

[cjnolet]

LGTM!

[cjnolet]

/merge