Skip to content

Commit

Permalink
Add Running a Workload (opensearch-project#6287)
Browse files Browse the repository at this point in the history 
* Add Running a Workload draft

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update running-workloads.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _benchmark/user-guide/running-workloads.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix link

Signed-off-by: Naarcha-AWS <[email protected]>

* Add additional missing link

Signed-off-by: Naarcha-AWS <[email protected]>

* Add running workloads

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Add numbered steps

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
  • Loading branch information
@Naarcha-AWS @vagimeli
@natebower @oeyh
3 people authored and oeyh committed Mar 14, 2024
1 parent b4a1526 commit 3a003db
Show file tree
Hide file tree
Showing 2 changed files with 169 additions and 1 deletion.
2 changes: 1 addition & 1 deletion _benchmark/user-guide/concepts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Before using OpenSearch Benchmark, familiarize yourself with the following conce

## Core concepts and definitions

- **Workload**: The description of one or more benchmarking scenarios that use a specific document corpus to perform a benchmark against your cluster. The document corpus contains any indexes, data files, and operations invoked when the workflow runs. You can list the available workloads by using `opensearch-benchmark list workloads` or view any included workloads in the [OpenSearch Benchmark Workloads repository](https://github.com/opensearch-project/opensearch-benchmark-workloads/). For more information about the elements of a workload, see [Anatomy of a workload](({{site.url}}{{site.baseurl}}/benchmark/understanding-workloads/anatomy-of-a-workload/). For information about building a custom workload, see [Creating custom workloads]({{site.url}}{{site.baseurl}}/benchmark/creating-custom-workloads/).
- **Workload**: The description of one or more benchmarking scenarios that use a specific document corpus to perform a benchmark against your cluster. The document corpus contains any indexes, data files, and operations invoked when the workflow runs. You can list the available workloads by using `opensearch-benchmark list workloads` or view any included workloads in the [OpenSearch Benchmark Workloads repository](https://github.com/opensearch-project/opensearch-benchmark-workloads/). For more information about the elements of a workload, see [Anatomy of a workload]({{site.url}}{{site.baseurl}}/benchmark/user-guide/understanding-workloads/anatomy-of-a-workload/). For information about building a custom workload, see [Creating custom workloads]({{site.url}}{{site.baseurl}}/benchmark/creating-custom-workloads/).

- **Pipeline**: A series of steps occurring before and after a workload is run that determines benchmark results. OpenSearch Benchmark supports three pipelines:
- `from-sources`: Builds and provisions OpenSearch, runs a benchmark, and then publishes the results.
Expand Down
168 changes: 168 additions & 0 deletions _benchmark/user-guide/running-workloads.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
---
layout: default
title: Running a workload
nav_order: 9
parent: User guide
---

# Running a workload

Once you have a complete understanding of the various components of an OpenSearch Benchmark [workload]({{site.url}}{{site.baseurl}}/benchmark/user-guide/understanding-workloads/anatomy-of-a-workload/), you can run your first workload.

## Step 1: Find the workload name

To learn more about the standard workloads included with OpenSearch Benchmark, use the following command:

```
opensearch-benchmark list workloads
```
{% include copy.html %}

A list of all workloads supported by OpenSearch Benchmark appears. Review the list and select the workload that's most similar to your cluster's use case.

## Step 2: Running the test

After you've selected the workload, you can invoke the workload using the `opensearch-benchmark execute-test` command. Replace `--target-host` with the `host:port` pairs for your cluster and `--client-options` with any authorization options required to access the cluster. The following example runs the `nyc_taxis` workload on a localhost for testing purposes.

If you want to run a test on an external cluster, see [Running the workload on your own cluster](#running-a-workload-on-an-external-cluster).

```bash
opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nyc_taxis --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false
```
{% include copy.html %}


Results from the test appear in the directory set by the `--output-path` option in the `execute-test` command.

### Test mode

If you want to run the test in test mode to make sure that your workload operates as intended, add the `--test-mode` option to the `execute-test` command. Test mode ingests only the first 1,000 documents from each index provided and runs query operations against them.

## Step 3: Validate the test

After running an OpenSearch Benchmark test, take the following steps to verify that it has run properly:

1. Note the number of documents in the OpenSearch or OpenSearch Dashboards index that you plan to run the benchmark against.
2. In the results returned by OpenSearch Benchmark, compare the `workload.json` file for your specific workload and verify that the document count matches the number of documents. For example, based on the [nyc_taxis](https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/nyc_taxis/workload.json#L20) `workload.json` file, you should expect to see `165346692` documents in your cluster.

## Expected results

OSB returns the following response once the benchmark completes:

```bash
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-------------------------------------------:|------------:|-------:|
| Cumulative indexing time of primary shards | | 0.02655 | min |
| Min cumulative indexing time across primary shards | | 0 | min |
| Median cumulative indexing time across primary shards | | 0.00176667 | min |
| Max cumulative indexing time across primary shards | | 0.0140333 | min |
| Cumulative indexing throttle time of primary shards | | 0 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0 | min |
| Cumulative merge time of primary shards | | 0.0102333 | min |
| Cumulative merge count of primary shards | | 3 | |
| Min cumulative merge time across primary shards | | 0 | min |
| Median cumulative merge time across primary shards | | 0 | min |
| Max cumulative merge time across primary shards | | 0.0102333 | min |
| Cumulative merge throttle time of primary shards | | 0 | min |
| Min cumulative merge throttle time across primary shards | | 0 | min |
| Median cumulative merge throttle time across primary shards | | 0 | min |
| Max cumulative merge throttle time across primary shards | | 0 | min |
| Cumulative refresh time of primary shards | | 0.0709333 | min |
| Cumulative refresh count of primary shards | | 118 | |
| Min cumulative refresh time across primary shards | | 0 | min |
| Median cumulative refresh time across primary shards | | 0.00186667 | min |
| Max cumulative refresh time across primary shards | | 0.0511667 | min |
| Cumulative flush time of primary shards | | 0.00963333 | min |
| Cumulative flush count of primary shards | | 4 | |
| Min cumulative flush time across primary shards | | 0 | min |
| Median cumulative flush time across primary shards | | 0 | min |
| Max cumulative flush time across primary shards | | 0.00398333 | min |
| Total Young Gen GC time | | 0 | s |
| Total Young Gen GC count | | 0 | |
| Total Old Gen GC time | | 0 | s |
| Total Old Gen GC count | | 0 | |
| Store size | | 0.000485923 | GB |
| Translog size | | 2.01873e-05 | GB |
| Heap used for segments | | 0 | MB |
| Heap used for doc values | | 0 | MB |
| Heap used for terms | | 0 | MB |
| Heap used for norms | | 0 | MB |
| Heap used for points | | 0 | MB |
| Heap used for stored fields | | 0 | MB |
| Segment count | | 32 | |
| Min Throughput | index | 3008.97 | docs/s |
| Mean Throughput | index | 3008.97 | docs/s |
| Median Throughput | index | 3008.97 | docs/s |
| Max Throughput | index | 3008.97 | docs/s |
| 50th percentile latency | index | 351.059 | ms |
| 100th percentile latency | index | 365.058 | ms |
| 50th percentile service time | index | 351.059 | ms |
| 100th percentile service time | index | 365.058 | ms |
| error rate | index | 0 | % |
| Min Throughput | wait-until-merges-finish | 28.41 | ops/s |
| Mean Throughput | wait-until-merges-finish | 28.41 | ops/s |
| Median Throughput | wait-until-merges-finish | 28.41 | ops/s |
| Max Throughput | wait-until-merges-finish | 28.41 | ops/s |
| 100th percentile latency | wait-until-merges-finish | 34.7088 | ms |
| 100th percentile service time | wait-until-merges-finish | 34.7088 | ms |
| error rate | wait-until-merges-finish | 0 | % |
| Min Throughput | percolator_with_content_president_bush | 36.09 | ops/s |
| Mean Throughput | percolator_with_content_president_bush | 36.09 | ops/s |
| Median Throughput | percolator_with_content_president_bush | 36.09 | ops/s |
| Max Throughput | percolator_with_content_president_bush | 36.09 | ops/s |
| 100th percentile latency | percolator_with_content_president_bush | 35.9822 | ms |
| 100th percentile service time | percolator_with_content_president_bush | 7.93048 | ms |
| error rate | percolator_with_content_president_bush | 0 | % |
[...]
| Min Throughput | percolator_with_content_ignore_me | 16.1 | ops/s |
| Mean Throughput | percolator_with_content_ignore_me | 16.1 | ops/s |
| Median Throughput | percolator_with_content_ignore_me | 16.1 | ops/s |
| Max Throughput | percolator_with_content_ignore_me | 16.1 | ops/s |
| 100th percentile latency | percolator_with_content_ignore_me | 131.798 | ms |
| 100th percentile service time | percolator_with_content_ignore_me | 69.5237 | ms |
| error rate | percolator_with_content_ignore_me | 0 | % |
| Min Throughput | percolator_no_score_with_content_ignore_me | 29.37 | ops/s |
| Mean Throughput | percolator_no_score_with_content_ignore_me | 29.37 | ops/s |
| Median Throughput | percolator_no_score_with_content_ignore_me | 29.37 | ops/s |
| Max Throughput | percolator_no_score_with_content_ignore_me | 29.37 | ops/s |
| 100th percentile latency | percolator_no_score_with_content_ignore_me | 45.5703 | ms |
| 100th percentile service time | percolator_no_score_with_content_ignore_me | 11.316 | ms |
| error rate | percolator_no_score_with_content_ignore_me | 0 | % |
--------------------------------
[INFO] SUCCESS (took 18 seconds)
--------------------------------
```



## Running a workload on an external cluster

Now that you're familiar with running OpenSearch Benchmark on a local cluster, you can run it on your external cluster, as described in the following steps:
1. Replace `https://localhost:9200` with your target cluster endpoint. This could be a Uniform Resource Identifier (URI), such as `https://search.mydomain.com`, or a `HOST:PORT` specification.
2. If the cluster is configured with basic authentication, replace the username and password in the command line with the appropriate credentials.
3. Remove the `verify_certs:false` directive if you are not specifying `localhost` as your target cluster. This directive is necessary solely for clusters without SSL certificates.
4. If you are using a `HOST:PORT`specification and plan to use SSL or TLS, either specify `https://` or add the `use_ssl:true` directive to the `--client-options` string option.
5. Remove the `--test-mode` flag to run the full workload rather than an abbreviated test.
You can copy the following command template to use it in your own terminal:
```bash
opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nyc_taxis --target-host=<OpenSearch Cluster Endpoint> --client-options=basic_auth_user:admin,basic_auth_password:admin
```
{% include copy.html %}

0 comments on commit 3a003db

Please sign in to comment.