Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial faiss interface support #42

Conversation

jmazanec15
Copy link
Member

@jmazanec15 jmazanec15 commented Jun 14, 2021

Description

This PR adds initial support through the knn_vector field type for faiss. This PR allows users to use HNSW method with flat encoding:

{
  "target-index" : {
    "mappings" : {
      "properties" : {
        "target-field" : {
          "type" : "knn_vector",
          "dimension" : 2,
          "method" : {
            "engine" : "faiss",
            "space_type" : "l2",
            "name" : "hnsw",
            "parameters" : {
              "m": 48,
              "encoder": {
                "name": "flat"
              }
            }
          }
        }
      }
    }
  }
}

In the future, we will add more methods.

The main changes include adding a MethodComponentContext parameter type, so that components such as encoders, may be configured through parameters, logic to build custom map of parameters including index description to be passed to faiss library, mapper interface changes, etc.

Additionally, added/refactored tests to better cover initial functionality.

Issues Resolved

#27

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@jmazanec15 jmazanec15 requested review from VijayanB and vamshin June 14, 2021 17:16
@jmazanec15 jmazanec15 changed the base branch from faiss-develop to feature/faiss-support June 24, 2021 17:00
Copy link
Member

@vamshin vamshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment. LGTM! Thanks

Copy link
Member

@VijayanB VijayanB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@jmazanec15 jmazanec15 merged commit c9a3566 into opensearch-project:feature/faiss-support Jun 29, 2021
jmazanec15 added a commit to jmazanec15/k-NN-1 that referenced this pull request Jun 30, 2021
commit c9a3566
Author: Jack Mazanec <[email protected]>
Date:   Tue Jun 29 15:38:39 2021 -0700

    Initial faiss interface support (opensearch-project#42)

    Signed-off-by: John Mazanec <[email protected]>

Signed-off-by: John Mazanec <[email protected]>
jmazanec15 added a commit to jmazanec15/k-NN-1 that referenced this pull request Oct 22, 2021
jmazanec15 added a commit that referenced this pull request Oct 22, 2021
martin-gaievski pushed a commit to martin-gaievski/k-NN that referenced this pull request Mar 7, 2022
martin-gaievski pushed a commit to martin-gaievski/k-NN that referenced this pull request Mar 7, 2022
Signed-off-by: Jack Mazanec <[email protected]>
Signed-off-by: Martin Gaievski <[email protected]>
martin-gaievski pushed a commit to martin-gaievski/k-NN that referenced this pull request Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants