Add vector deduplication for HNSW

[heemin32]

Add vector deduplication for HNSW

[mdouze]

Thanks for doing the effort of implementing this deduplication functionality.
However, as I said in #3087 I don't think it is of sufficient interest.
We are trying to limit the amount of code in Faiss because otherwise it is more work to maintain.

[heemin32]

@mdouze Thanks for the comment. Could you tell how to decide if there are sufficient interest or not? OpenSearch uses faiss for knn search engine and there are many interest on document deduplication feature. In other words, what would make your mind change to accept this PR?

[cjnolet]

I don't want to take any attention away from FAISS here, but multi-valued keys have been requested several times in the RAFT / cuVS library. It's on our roadmap to hopefully implement sometime within the year. If the feature ends up performing well, we would be interested in contributing it to FAISS. At the very least, we could write up an example of how to accomplish this functionality using the Boolean filter function.