Adding support for binary embeddings

[bhugueney]

Thank you for this most useful extension !
It seems that binary embeddings allow dramatic increase in performance for a small accuracy cost ( https://huggingface.co/blog/embedding-quantization#quantization-experiments ).
Various other vector DB allow to use them :

• Faiss Yes

• USearch Yes

• Vespa AI Yes

• Milvus Yes

• Qdrant Through Binary Quantization

• Weaviate Through Binary Quantization

It would be great if DuckDB vss could also support them efficiently.

EDIT pg_vecor also has it : https://github.com/pgvector/pgvector?tab=readme-ov-file#binary-vectors
Best Regards

[Maxxen]

Hello!

Yes, support for arrays of other types is planned, although binary vectors in particular might be slightly more complex since DuckDB itself doesn't really have a "bit" type, but it should be doable.

[bhugueney]

Hello!

Yes, support for arrays of other types is planned, although binary vectors in particular might be slightly more complex since DuckDB itself doesn't really have a "bit" type, but it should be doable.

Thank you for your interest.
I have high hopes for the binary vector as they provide considerable saving in memory and speed both because of data throughput (cf. memory) and simpler distance implementation
https://github.com/CountOnes/hamming_weight.
Best Regards.

Sorry, closed by mistake and couldn't see how to undo the closing.