Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Workload Improvement: Multi-Process Index Extraction #375

Open
IanHoang opened this issue Sep 25, 2023 · 1 comment
Open

Create Workload Improvement: Multi-Process Index Extraction #375

IanHoang opened this issue Sep 25, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@IanHoang
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

When using OpenSearch Benchmark's create-workload feature to extract data from clusters, users often extract more than one index. For example, a user might have a cluster comprised of 5 indices and OSB will extract each corpora in order. This is a time consuming process and can be more efficient. For larger sets of data, this can also lead to read timeouts like below:

[ERROR] Cannot create-workload. ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='<example host endpoint>', port=443): Read timed out. (read timeout=10)).

Describe the solution you'd like

To expedite the process and make it more efficient, OSB can use N number of processes to extract N number of indices.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

@AkshathRaghav
Copy link
Contributor

@IanHoang 👋🏽, I've opened a PR for this -> #403

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants