ANN_BENCH

[achirkin]

Porting the ANN benchmarks from RAFT.

• Make it build

Sanity check that benchmarks work (runs and gives reasonable recall for Deep-1M dataset)

• cuVS brute force kNN
• cuVS IVF-Flat
• cuVS IVF-PQ (+ refinement)
• cuVS CAGRA
• cuVS CAGRA-Q (+refinement)
• Faiss GPU/CPU IVF-Flat & IVF-PQ
• HNSW
• CAGRA + HNSW
• GGNN

NB: the indices built using the old ANN_BENCH in raft tend to crash in cuvs search benchmarks during index deserialization - don't forget to build the indexes anew when testing.

[achirkin]

I've just realized the benchmarks are not compiled during conda-cpp-build CI. @cjnolet what's the best way to add the benchmark component to CI build?

[achirkin]

NB: some algorithms are likely to fail with OOM thrown by the limiting_resource_adapter with very large datasets (e.g. DEEP-1B); the fix is in #181

[achirkin]

I tried to add benchmarks to the test build in CI, but got the error from faiss not being able to find BLAS libraries, even though I added openblas as a conda dependency. @cjnolet, @benfred, could you please have a look?

[tfeher]

Thanks Artem for the updates. This PR ports the existing infrastructure from raft and enables us to run all the existing the benchmarks. I believe it would be useful to have this merged, and work o follow up improvements in separate PRs. I have added open discussion points to the tracker issue #160 (comment).

The PR looks good to me. I have also contributed to the PR, so I shall not be the only approver.

[jameslamb]

I've reviewed on behalf of packaging-codeowners. Left 2 small comments, neither needs to block merging this.

[jameslamb]

Over in RAFT, bench-ann is its own package with its own dependencies, build scripts, etc.:

https://github.com/rapidsai/raft/tree/877644a423c0268746af62cecb7150afa65d8386/python/raft-ann-bench

https://github.com/rapidsai/raft/blob/877644a423c0268746af62cecb7150afa65d8386/conda/recipes/raft-ann-bench/meta.yaml

For my own understanding in reviewing this... why is this being added to the libcuvs-tests package here instead of creating a new standalone package as exists in RAFT?

Having these separate things be their own packages can be helpful for parallelizing development, limiting the potential impact of packaging changes, and speeding up debugging.

[achirkin]

I agree that benchmarks should probably be a separate package. I'm only hesitating to add this because I'm not very familiar with this conda/CI setup. I hoped to postpone this question till a follow-on PR together with the naming decision (#160 (comment)). @benfred, @cjnolet , what do you think?

[benfred]

Can we add an option to build this in CI?

It looks to me that this is off by default (which is fine), but that does mean that we don't have any guarantee that any of this is compiling.

[achirkin]

We enable this currently together with tests in build.sh (see #130 (comment))

[cjnolet]

/merge