Tracking issue for schema

[usamoi]

View https://docs.pgvecto.rs/reference/schema.html to see schema provided by pgvecto.rs.

Functions

pgvecto.rs are lack of helper functions, and we need to provide more helper functions.

• util functions
  • function vector_dims(vector) RETURNS int, get dimensions of a vector feat: support aggregate function for vector. #463 feat: add vector_norm, vector_dims and normalize for all vector types #485
  • function normalize(vector) RETURNS vector, normalize a vector feat: add vector_norm, vector_dims and normalize for all vector types #485
  • function vector_norm(vector) RETURNS real, $L^2$ norm of a vector feat: support aggregate function for vector. #463 feat: add vector_norm, vector_dims and normalize for all vector types #485
  • aggregate function avg(vector) and sum(vector) feat: support aggregate function for vector. #463
• casts
  • casts between vector -> vecf16: feat: Function / cast to convert vectors into vecf16 #262 feat: Add vector to vecf16 CAST #266
  • casts between vecf16 -> vector, vector -> svector, svector -> vector: feat: support sparse vector #299
  • casts between vecf16 and half precision[]: PostgreSQL do not support half precision[], we may make some decisions here not supported
• embedding functions
  • openai: feat: new openai embedding #350 docs: document pgvectors_upgrade and text2vec_openai in schema pgvecto.rs-docs#56

Types

• text representation of vector types, not documented
• memory representation of vector types, not documented
• binary representation of vector types, not documented
• subscript behaviors of vector types, not documented

Schema

• install extension with schema bug: install at schema would fail at create index #270

CI

• check that you write a CREATE FUNCTION export_name SQL using the name _vectors_*_wrapper and the SQL is inconsistent with the signture defined in the code
• check if a correct install script is located in ./src/install for a release

Contributing

If you want to export a user function to PostgreSQL:

1. use pgrx::pg_extern and name the function _vectors_*
2. write CREATE FUNCTION export_name in finalize.sql, you can use the name _vectors_*_wrapper here.

If you want to export a cast, operator or something else to PostgreSQL:

1. use pgrx::pg_extern and name the function _vectors_*
2. finish SQL in finalize.sql

[VoVAllen]

Test logical replication between different extension version

[JoePassanante]

Can you elaborate on

casts between vecf16 and half precision; PostgreSQL do not support half precision, we may make some decisions here

Is this an issue with postgres itself? or that it hasn't been implemented in pgvecto?

[usamoi]

Is this an issue with postgres itself?

Yes. PostgreSQL supports floating-type real (IEEE 754 binary 32) and floating type double precision (IEEE 754 binary 64), but there is not such a type named half precision (IEEE 754 binary 16). We probably cannot provide a cast between a vecf16 and an array. Another choice is providing a cast between vef16 and real[], which is "make some decisions here" refered.

[JoePassanante]

Is this an issue with postgres itself?

Yes. PostgreSQL supports floating-type real (IEEE 754 binary 32) and floating type double precision (IEEE 754 binary 64), but there is not such a type named half precision (IEEE 754 binary 16). We probably cannot provide a cast between a vecf16 and an array. Another choice is providing a cast between vef16 and real[], which is "make some decisions here" refered.

Messaged over in the discord for some help getting the environment setup. But isn't this as simple as:

#[pgrx::pg_extern(immutable, parallel_safe, strict)]
fn _vectors_cast_vecf32_to_vecf16(
    vector: Vecf32Input<'_>,
    _typmod: i32,
    _explicit: bool,
) -> Vecf16Output {
    let data = vector
        .data()
        .iter()
        .map(|x| x.to_f32())
        .map(|x| f16::from_f32(x))
        .collect();

    Vecf16::new_in_postgres(data)
}

CREATE CAST (vector AS vecf16)
    WITH FUNCTION _vectors_cast_vecf32_to_vecf16(vector, integer, boolean) AS IMPLICIT;

[usamoi]

Is this an issue with postgres itself?

Yes. PostgreSQL supports floating-type real (IEEE 754 binary 32) and floating type double precision (IEEE 754 binary 64), but there is not such a type named half precision (IEEE 754 binary 16). We probably cannot provide a cast between a vecf16 and an array. Another choice is providing a cast between vef16 and real[], which is "make some decisions here" refered.

Messaged over in the discord for some help getting the environment setup. But isn't this as simple as:

#[pgrx::pg_extern(immutable, parallel_safe, strict)]
fn _vectors_cast_vecf32_to_vecf16(
    vector: Vecf32Input<'_>,
    _typmod: i32,
    _explicit: bool,
) -> Vecf16Output {
    let data = vector
        .data()
        .iter()
        .map(|x| x.to_f32())
        .map(|x| f16::from_f32(x))
        .collect();

    Vecf16::new_in_postgres(data)
}

CREATE CAST (vector AS vecf16)
    WITH FUNCTION _vectors_cast_vecf32_to_vecf16(vector, integer, boolean) AS IMPLICIT;

Yes.

• casts between vector and vecf16
• casts between vecf16 and half precision[]

It's two issues. You can talk only about the first one since you only need the first one.

[VoVAllen]

I think float(2) is the same as fp16?

[usamoi]

I think float(2) is the same as fp16?

https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-FLOAT

I don't think so.

[my-vegetable-has-exploded]

Is there more specific definition of absolute-value norm?

[usamoi]

Is there more specific definition of absolute-value norm?

https://en.wikipedia.org/wiki/Norm_(mathematics)

[my-vegetable-has-exploded]

Is there more specific definition of absolute-value norm?

https://en.wikipedia.org/wiki/Norm_(mathematics)

I did not grasp the definition. How should we calculate it for a one-dimensional vector?