Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validation for PQ based methods based on dimension #1075

Closed
jmazanec15 opened this issue Aug 30, 2023 · 0 comments
Closed

Add validation for PQ based methods based on dimension #1075

jmazanec15 opened this issue Aug 30, 2023 · 0 comments
Assignees
Labels
Enhancements Increases software capabilities beyond original client specifications v2.15.0

Comments

@jmazanec15
Copy link
Member

Related to #1074 (comment), right now, when validating a method to be used for training for PQ, we do not validate that the field's dimension is in fact divisble by the M parameter.

PQ encodes vectors by breaking them into M sub-vectors and then encoding each sub-vector independently. The behavior for the user will be that they get a vague excpetion. Something like: https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/training/TrainingJob.java#L196.

Instead, we should give a more descriptive error message that says exactly this. It will require some refactoring to https://github.com/opensearch-project/k-NN/blob/2.9.0.0/src/main/java/org/opensearch/knn/index/util/Faiss.java#L95

@jmazanec15 jmazanec15 added bug Something isn't working Enhancements Increases software capabilities beyond original client specifications and removed untriaged bug Something isn't working labels Aug 30, 2023
@vamshin vamshin moved this from Backlog to 2.15.0 in Vector Search RoadMap May 14, 2024
@github-project-automation github-project-automation bot moved this from 2.15.0 to ✅ Done in Vector Search RoadMap May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancements Increases software capabilities beyond original client specifications v2.15.0
Projects
Status: Done
Development

No branches or pull requests

3 participants