embedding_vector_search

prototyping global search for embedding vectors

Introduction

The objective of this repo is to prototype a global search tool for embedding vectors of the scale of roughly:

• 1-10M vectors
• 200-2k dimensions
• Top 10 results, with recall of ???
• SLA < 1 sec

Datasets

For testing, we use the following strategy

Synthetic datasets

Generate random vectors using scikit-learn, see examples in https://github.com/erikbern/ann-benchmarks/blob/14df47df72bb1deee5d9e68616ba03edf65d4680/ann_benchmarks/datasets.py#L473

Parameters:

Open source datasets

See https://github.com/erikbern/ann-benchmarks for available test vectors, in HDF5 format. Choose some that are closer to our needs.

Proprietary datasets

We can generate our own embedding vectors.

Loading/Enrolling

We need to load the embedding vectors to a proper data structure, either in memory or in a database. For the application we have in mind, we also need to link the vectors to some metadata, presummably in a JSON object.

Search

We conduct a 10-NN search over the full list that is filtered based on the metadata.