Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Fix legacy issue in index #553

Merged
merged 7 commits into from
Jan 13, 2023
Merged

feat: Fix legacy issue in index #553

merged 7 commits into from
Jan 13, 2023

Conversation

jiashenC
Copy link
Member

@jiashenC jiashenC commented Jan 5, 2023

  • Remove secondary index maintained by ourselves. Instead, we now use IDMap provided by the FAISS library, which is also faster in terms of performance.
  • Add support of creating index on UDF expression directly. Along with that, add udf signature as part of the index catalog, so the expression can be matched again when index is used later.

@jiashenC jiashenC changed the title Fix legacy issue in index feat: Fix legacy issue in index Jan 5, 2023
@jiashenC jiashenC requested a review from gaurav274 January 5, 2023 19:11
@xzdandy
Copy link
Collaborator

xzdandy commented Jan 8, 2023

I'd like to get some context on adding support of creating index on UDF expression directly. For this case CREATE INDEX ON table (FeatureExtractor(col));, what is the expected index we want to create? Do we treat FeatureExtractor(col) as a table and create index on the feature extracted and store the mapping between the index and row_id?

@jiashenC
Copy link
Member Author

jiashenC commented Jan 8, 2023

I'd like to get some context on adding support of creating index on UDF expression directly. For this case CREATE INDEX ON table (FeatureExtractor(col));, what is the expected index we want to create? Do we treat FeatureExtractor(col) as a table and create index on the feature extracted and store the mapping between the index and row_id?

Yes, you are right.

@xzdandy
Copy link
Collaborator

xzdandy commented Jan 9, 2023

I'd like to get some context on adding support of creating index on UDF expression directly. For this case CREATE INDEX ON table (FeatureExtractor(col));, what is the expected index we want to create? Do we treat FeatureExtractor(col) as a table and create index on the feature extracted and store the mapping between the index and row_id?

Yes, you are right.

Thanks. Overall LGTM.

@jiashenC jiashenC force-pushed the fix-legacy-issue-in-index branch from 6ab158d to e975960 Compare January 12, 2023 15:54
@jiashenC jiashenC requested a review from xzdandy January 12, 2023 16:28
@jiashenC jiashenC merged commit a05d84d into master Jan 13, 2023
@jiashenC jiashenC deleted the fix-legacy-issue-in-index branch January 13, 2023 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants