Add YAML config files to run parameter sweeps for ANN benchmarks #1929
Conversation
cjnolet
added
improvement
There was a problem hiding this comment.
I haven't gone through the readme yet but so far everything else looks good. Mostly minor things in the help guides.
I'll review the docs when ready. Please also remember we need a copy/paste example of what a custom yaml file should look like. It doesn't have to be real, and doesn't have to include the validators since those should always be loaded first in the included configs. At least something that shows multiple groups an the association between a couple parameter names and values (for both build and search)
python/raft-ann-bench/src/raft-ann-bench/validators/__init__.py
Outdated
Show resolved
Hide resolved
docs/source/raft_ann_benchmarks.md
Outdated
|
|
||
| The table below contains the possible settings for the `algo` field. Each unique algorithm will have its own set of `build_param` and `search_params` settings. The [ANN Algorithm Parameter Tuning Guide](ann_benchmarks_param_tuning.md) contains detailed instructions on choosing build and search parameters for each supported algorithm. | ||
| There's several things going on here: | ||
| 1. `groups` - define a run group which has a particular set of parameters. Each group has `build` parameters wherein a cross-product of all parameters will be used to build a unique index. Each group also has `search` parameters, and a cross-product of all search parameters will be used to search all unique indices that we create. |
There was a problem hiding this comment.
"cross-product of all hyper-parameter fields for build and search".
Can you please link out to the parameter tuning guide and mention that some parameters in the tuning guide are required, but otherwise don't need to be specified.
There was a problem hiding this comment.
The tuning guide is linked in the sentence below
…g `dim` to datasets.yaml. Adding new yaml configs for FAISS. Fixing yaml config for RAFT
…nto fea-2312-bench-ann-conf
Also generate CUDA 12.0 development environments for
bench-ann