[DOC] Add efficient filtering capability for Lucene knn

[martin-gaievski]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.

k-NN plugin in OpenSearch will support filtering. This will allow users to define custom filters as part of the search request for knn_vector data type. Filtering functionality will be supported for case when Lucene has been defined as an engine for hnsw method.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

opensearch-project/k-NN#376

https://github.com/martin-gaievski/k-NN/tree/poc/pre-filtering-for-lucene-engine

[alicejw1]

10/24 Status: started research, gathered all resource specs and recording of demo. Started writing 10/24.

[alicejw1]

Update: writing content, on-target for tech review of two drafts by 11/3 this week.

[dtaivpp]

@alicejw-aws and @martin-gaievski can we remove these quip links and either past the content in a comment or post it somewhere public? Thanks!

[alicejw1]

hi @dtaivpp, will do! I've saved the resources for doc purposes, and removed the quip links to this issue. Not sure if they are made public or not. Thanks

[alicejw1]

11-7 update: doc is in eng review with Martin and his team.
New section is in-prog to be added for outlining the top 3 use cases, and add an algorithm diagram image with a table of three types of filters that vary from very selective (2%) to medium (40%) to very selective (80%).

[alicejw1]

Update: Tech review is completed 11-8, now in doc review.