Product Quantization on HNSW(HNSW+PQ) in FAISS Engine

[jmazanec15]

Coming from https://forum.opensearch.org/t/cannot-create-index-with-hnsw-pq/15634/1 (@tomekcejner-cloudinary).

{
    "training_index": "training-data-index",
    "training_field": "embedding_vector",
    "dimension": 512,
    "description": "HNSW with PQ",
    "method": {
        "name": "hnsw",
        "engine": "faiss",
        "space_type": "l2",
        "parameters": {
            "encoder": {
                "name": "pq",
                "parameters": {
                    "code_size": 8,
                     "m": 8
                }
            }
        }
    }
}

[2023-08-23T21:17:09,996][ERROR][o.o.k.t.TrainingJob      ] [fa696978b96b] Failed to run training job for model "hnsw_pq_m8": Error in std::unique_ptr<faiss::Index> faiss::{anonymous}::index_factory_sub(int, std::string, faiss::MetricType) at /tmp/tmpzmr1s8yb/k-NN/jni/external/faiss/faiss/index_factory.cpp:781: Error: 'index' failed: could not parse HNSW code description PQ8x8 in HNSW16,PQ8x8

HNSWPQ does not work in OpenSearch because the faiss index factory description constructed for HNSW is "HNSW16,PQ1x8", which is not valid due to nbits not being parametrizable for HNSW in faiss. See facebookresearch/faiss#3027. Will follow up on discussion there. To fix in short term, we need to change https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/util/Faiss.java#L104 to format selectively based on base index type

[DarshitChanpura]

@jmazanec15 Should this be labelled to 2.11?

[navneet1v]

@DarshitChanpura this is already merged in 2.10 and will be released with 2.10