[Tech Debt] Improve build time

[wphicks]

Summary

It would be useful to improve build time as much as possible for faster developer iteration and reduced CI resource consumption.

Current bottleneck files

The following are roughly in order of priority. Note that this does not correspond strictly to the longest build times (although it mostly does). Instead, it takes into account what can and can't be parallelized and ensures that we're tackling each dependent branch.

• silhouette_score.cu: far and away our worst offender, with a compile duration of 22 minutes on my local machine. Fixing this alone could cut our compile time by a third
• knn_regress_mg.cu
• hdbscan.cu
• tsne.cu
• fil/infer.cu: Will be resolved if Provide FIL implementation for both CPU and GPU #4890 replaces existing implementation
• hierarchy/linkage.cu
• knn_classify_mg.cu
• umap.cu
• knn.cu
• tree_shap.cu
• trustworthiness.cu
• svc.cu
• knn_mg.cu
• svr.cu

[wphicks]

Note, this issue replaces the original #3501 since there has been a fair amount of movement since then on this problem.

[cjnolet]

Some of these things, especially those things which require pairwise distances, will likely benefit from using the pre-compiled specializations that are already being built in RAFT. We've scraped through most things and updated them but there are still some things lingering, unfortunately, which might need to be updated (thinking single linkage, trustworthiness, silhouetter_score, hdbscan)

[wphicks]

Just submitted #5061 to get the low-hanging fruit, but silhouette score remains our most significant bottleneck.