Support for ANN Indexing in Nested k-NN Vectors in OpenSearch

[KreshLaDoge]

Is your feature request related to a problem? Please describe.
I'm encountering limitations when trying to perform efficient approximate k-NN searches on nested fields containing k-NN vectors in OpenSearch. Currently, the k-NN plugin doesn't support ANN indexing for vectors within nested fields, leading to less efficient searches, particularly in large datasets with high dimensionality.

Describe the solution you'd like
I would like the k-NN plugin to support ANN indexing for k-NN vectors within nested fields. This feature would enable more efficient and scalable searches by leveraging the ANN algorithms already integrated into OpenSearch, such as those from nmslib, faiss, and Lucene.

Describe alternatives you've considered

1. Flattening the data structure, though this requires significant changes to the data model and might not be feasible for all use cases.
2. Using external ANN indexing systems alongside OpenSearch, which adds complexity to my system overall and redundancy of some data.
3. Employing a custom script for brute-force search, which is resource-intensive and less scalable.

Additional context
Efficient k-NN search in nested structures is crucial for many applications dealing with complex data models, where vectors are naturally nested within documents. Enhancing the k-NN plugin to support ANN indexing in such scenarios would significantly improve the usability and performance of k-NN searches in OpenSearch, especially for large-scale, high-dimensional datasets.

[dblock]

Moved this to k-nn.

[heemin32]

@KreshLaDoge Thanks for the feature request. Could you +1 in this facebook post and also in this github issue so that the feature can be supported in faiss engine?

[jmazanec15]

I think this is covered in #1398.