Implement the Vector of Floats data type and basic vector search scalar functions

[eolivelli]

Motivation:

In Generative AI Application having the ability to deal with vectors (of float) is fundamental in order to implement the RAG patten.

While HerdDB is not meant to be a Vector database, it is still true the one of the HerdDB goals is to be a small footprint embeddable database, very useful for testing and for small applications.

Contents of the patch:

• This patch introduces end to end support for the "Float array" data type, both in the core (server side) and in the JDBC Driver
• We are adding 3 basic scalar functions to compare vectors of floats: cosine_similarity, dot_product and euclidean_distance

In order to support the RAG pattern you only need 2 features:

• inserting records with a vector field
• retrieve the records ordering using some "distance" function

This patch doesn't add any special index, it is really out of the scope of HerdDB, but with this key features it is pretty easy to build a small vector database to index thousands of documents and support chat bots.

Sample table:

CREATE TABLE DOCUMENTS (
            FILENAME string
            CHUNKID int, "
            TEXT string, "
             EMBEDDINGSVECTOR floata,
             PRIMARY KEY(filename, chinkid)
)

Sample Vector Search query

SELECT text
FROM DOCUMENTS
ORDER BY cosine_similarity(embeddingsvector,cast(? as FLOAT ARRAY)) DESC
LIMIT 10

[aluccaroni]

+1