Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] k-NN plugin: add reference to hybrid mmap extensions setting under performance tunning #2886

Closed
1 of 4 tasks
martin-gaievski opened this issue Feb 14, 2023 · 0 comments · Fixed by #3262
Closed
1 of 4 tasks
Assignees
Labels
1 - Backlog Issue: The issue is unassigned or assigned but not started v-All This issue is valid for all versions through 1.3

Comments

@martin-gaievski
Copy link
Member

What do you want to do?

  • Request a change to existing documentation
  • Add new documentation
  • Report a technical problem with the documentation
  • Other

Tell us about your request.
We need to add description of a performance tuning of the k-NN plugin. Possible place is a new section under https://opensearch.org/docs/latest/search-plugins/knn/performance-tuning/#search-performance-tuning.

Content can be something along these lines:

Use mmap file I/O

For approximate k-NN search that is based on Lucene engine there is no special cache layer that speeds up read/write operations, plugin is relying on existing mechanism in OpenSearch core. By default in versions 2.4 and below such operations were based on Java NIO, that can be slow depending on the Lucene version and number of segments per shard.
Starting from version 2.5 k-NN enables mmap file I/O by default when store type is hybridfs (this is default store type in OpenSearch). This enables fast file I/O operations and improves overall performance of both data ingestion and search. Two file extensions specific to vector values are marked for mmap, ".vec" and ".vem", please check Lucene documentation for details of each extensions.

MMap uses system file cache rather than memory allocated for Java heap, no additional allocation is required. In case it's required to change default list of extensions set by the plugin, please update index.store.hybrid.mmap.extensions setting at the cluster level using Cluster API. Please note, this is an expert level setting, index needs to be closed before update, and the re-open.

What other resources are available?
Cx request that triggered investigation and change: opensearch-project/k-NN#637

@Naarcha-AWS Naarcha-AWS added 1 - Backlog Issue: The issue is unassigned or assigned but not started v-All This issue is valid for all versions through 1.3 Priority: medium and removed untriaged Priority: medium labels Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 - Backlog Issue: The issue is unassigned or assigned but not started v-All This issue is valid for all versions through 1.3
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants