Case studies

In this page, we reference example use cases for Faiss, with some explanations. The examples will most often be in the form of Python notebooks, but as usual translation to C++ should be smooth.

Implementing an evolving IVF dataset

This script demonstrates how to add/remove elements from an IVF dataset in a rolling fashion. The key is to use a Hashtable as DirectMap type and remove with IDSelectorArray. Removal cost is then proportional to the number of elements to remove instead of number of elements in the dataset.

demo_rolling_dataset.ipynb

Fast indexing of 2M vectors for max inner product search

This script demonstates how to speed up a recommendation system. Conceptually, the queries vectors are users and the database vectors are items to recommend. The metric to "compare" them is maximum inner product, ie. which item is the most relevant for each user. There is a real-time constraint for this use case (should be returned in < 5 ms) and the accuracy should be as high as possible.

recommendation_2M.ipynb

Limited size clustering

This script demonstrates how to do a k-means variant where in addition the clusters are constrained to contain no more than a maximum number of points.

limited_size_clustering.ipynb

Asymmetric binary search

This script demonstrates an asymmetric search use case: the query vectors are in full precision and the database vectors are compressed as binary vectors. This implementation is slow, it is mainly intended to show how much accuracy can be regained with asymmetric search.

demo_asymmetric_binary.ipynb

Manual training of IVFPQ

This script demonstrates how to manually train an IVFPQ index enclosed in a OPQ pre-processor. This can be useful, for example, if there are pre-trained centroids handy for the data distribution.

This is also implemented in the function train_ivf_index_with_2level. It should be easy to expand to other types of composite indexes.

manual_IVFPQ_training.ipynb

Mixed sparse-dense clustering

There is a sparse clustering implementation in faiss.contrib.clustering. This script demonstrates how to cluster vectors that are composed of a dense part of dimension d1 and a sparse part of dimension d2 where d2 >> d1. The centroids are represented as full dense vectors.

The implementation relies on the clustering.DatasetAssign object, that abstracts away the representation of the vectors to cluster. The clustering module contains a pure Python implementation of kmeans that can consume this DatasetAssign.

sparse_dense_clustering.ipynb

Separate coarse quantization

This script demonstrates how to use a high-dimensional coarse quantizer with a low-dimensional fine quantizer. This is not possible out-of-the-box because the IVFPQ implementation assumes the IVF quantizer and the PQ run in the same dimension. To combine the quantizers in different dimensionalities, the approach is to use search_preassigned and add_preassigned to perform the coarse quantization and add / search separately.

In this example, the OPQ pre-transformation (an orthonormal transformation of the data) reduces the dimension of the input from 96 to 32 dimensions, so the coarse quantizer may not be as selective as it could. By doing the coarse quantization and the search separately, the accuracy improves for some (but not all) settings of nprobe.

demo_independent_ivf_dimension.ipynb

Manual implementation of IVFPQ search

This script demonstrates how to peform IVFPQ search in Python manually.

demo_ivfpq_distance_tables.ipynb

Sharded GPU dataset with simultaneous queries (pytorch)

This example script shows how to handle a database sharded over n GPUs. Each GPU issues a set of queries simultaneously. The queries are performed over the sharded dataset and the results are sent back to the issuing GPU. This is typical for pytorch training jobs, that need to do searches at each iteration over a dataset that is scattered around the training GPUs.

bench_faiss_n2n.py