[FEATURE] Support pre-filter queries for Lucene-based approximate k-NN

[vamshin]

Is your feature request related to a problem?
We do not have support to pre filter docs and then apply k-NN specifically on the filtered docs. This will help guarantee to obtain k Nearsest Neighbors on the filtered set. Please note, this is applicable only to Lucene based approximate k-NN search

What solution would you like?
Rely on Lucene's KnnVectorQuery to operate over a subset of liveDocs.
https://issues.apache.org/jira/browse/LUCENE-10382

[KelseyJuraschka]

I would love to see this feature implemented. Any ideas on when in the roadmap this may land? Elastic recently released their implementation.

[vamshin]

Hi @KelseyJuraschka we are targeting 2.3 release https://github.com/orgs/opensearch-project/projects/1#card-81105162

[ngadkari]

For my understanding this to add pre-filtering support for KNN indices with Lucene 'engine' support. Are there similar work to support this for nmslib and fais engine

[ymartin-mw]

@martin-gaievski Hey
Does the feature for pre-filtering on lucene hnsw ANN k-NN released in Opensearch 2.4?
Any estimation or knowledge how to upgrade AWS managed Opensearch from the maximum available 2.3 to 2.4?

Thanks

[navneet1v]

Yes this feature is released in OpenSearch 2.4.

Any estimation or knowledge how to upgrade AWS managed OpenSearch from the maximum available 2.3 to 2.4?

On this it will better if you can directly ask support team of AWS managed OpenSearch. As per my knowledge AWS managed OpenSearch will not support 2.4 version and they will directly launch 2.5. Again it will be better if you can contact AWS support team directly.

[martin-gaievski]

@ymartin-mw 2.5+ is better option for upgrade, in 2.4 lucene version 9.4 bring certain performance problems related to new way of doing segment merge in background. They are fixed in 2.5 (check #637 and #721 for details)

[ymartin-mw]

Just got a message from AWS that they released the OpenSearch 2.5 version 🎉

[ymartin-mw]

Any reference on how to build such query that does pre-filtering before approximate k-NN (lucene, hnsw)?

[navneet1v]

@ymartin-mw Please look into this documentation: https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/