Add ANN bench scripts to generate ground truth

[tfeher]

Add a command line based tool that can read a dataset and generate ground truth files with exact neighbors using raft brute force knn.

[tfeher]

usage: generate_groundtruth [-h] [--queries QUERIES] [--output OUTPUT] [--n_queries N_QUERIES] [-N ROWS] [-D COLS] [--dtype DTYPE] [-k K] [--metric METRIC] dataset

Generate true neighbors using exact NN search. The input and output files are in big-ann-benchmark's binary format.

positional arguments:
  dataset               input dataset file name

options:
  -h, --help            show this help message and exit
  --queries QUERIES     Queries file name, or one of 'random-choice' or 'random' (default). 'random-choice': select n_queries vectors from the input dataset. 'random': generate n_queries as uniform random numbers.
  --output OUTPUT       output directory name (default current dir)
  --n_queries N_QUERIES
                        Number of quries to generate (if no query file is given). Default: 10000.
  -N ROWS, --rows ROWS  use only first N rows from dataset, by default the whole dataset is used
  -D COLS, --cols COLS  number of features (dataset columns). Default: read from dataset file.
  --dtype DTYPE         Dataset dtype. When not specified, then derived from extension. Supported types: 'float32', 'float16', 'uint8', 'int8'
  -k K                  Number of neighbors (per query) to calculate
  --metric METRIC       Metric to use while calculating distances. Valid metrics are those that are accepted by pylibraft.neighbors.brute_force.knn. Most commonly used with RAFT ANN are 'sqeuclidean' and 'inner_product'

Example usage
    # With existing query file
    python -m raft-ann-bench.generate_groundtruth --dataset /dataset/base.fbin --output=groundtruth_dir --queries=/dataset/query.public.10K.fbin

    # With randomly generated queries
    python -m raft-ann-bench.generate_groundtruth --dataset /dataset/base.fbin --output=groundtruth_dir --queries=random --n_queries=10000

    # Using only a subset of the dataset. Define queries by randomly
    # selecting vectors from the (subset of the) dataset.
    python -m raft-ann-bench.generate_groundtruth --dataset /dataset/base.fbin --nrows=2000000 --cols=128 --output=groundtruth_dir --queries=random-choice --n_queries=10000
 

[cjnolet]

This is going to be very useful to have in the raft-ann-bench code, thanks for adding! Hopefully soon we will also be able to provide a means for users to convert their own datasets into the proper binary format to be loaded by raft-Ann-bench also.

[tfeher]

Thanks Corey for the review, I have updated the PR accordingly.

[tfeher]

Thanks @cjnolet for the second review, fixed the issues.

[cjnolet]

LGTM!

[cjnolet]

/merge