forked from opensearch-project/documentation-website
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Benchmark command line reference (opensearch-project#4173)
* Add Benchmark command reference Signed-off-by: Naarcha-AWS <[email protected]> * Add initial Benchmark command line reference Signed-off-by: Naarcha-AWS <[email protected]> * Address technical feedback Signed-off-by: Naarcha-AWS <[email protected]> * More technical feedback Signed-off-by: Naarcha-AWS <[email protected]> * More technical feedback Signed-off-by: Naarcha-AWS <[email protected]> * Fix list description Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Chris Moore <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Chris Moore <[email protected]>
- Loading branch information
1 parent
52188ae
commit 454c4f7
Showing
7 changed files
with
628 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| --- | ||
| layout: default | ||
| title: compare | ||
| nav_order: 55 | ||
| parent: Command reference | ||
| --- | ||
|
|
||
| # compare | ||
|
|
||
| The `compare` command helps you analyze the difference between two benchmark tests. This can help you analyze the performance impact of changes made from a previous test based on a specific Git revision. | ||
|
|
||
| ## Usage | ||
|
|
||
| You can compare two different workload tests using their `TestExecution IDs`. To find a list of tests run from a specific workload, use `opensearch-benchmark list test_executions`. You should receive an output similar to the following: | ||
|
|
||
|
|
||
| ``` | ||
| ____ _____ __ ____ __ __ | ||
| / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ | ||
| / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ | ||
| / /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< | ||
| \____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| | ||
| /_/ | ||
| Recent test-executions: | ||
| Recent test_executions: | ||
| TestExecution ID TestExecution Timestamp Workload Workload Parameters TestProcedure ProvisionConfigInstance User Tags workload Revision Provision Config Revision | ||
| ------------------------------------ ------------------------- ---------- --------------------- ------------------- ------------------------- ----------- ------------------- --------------------------- | ||
| 729291a0-ee87-44e5-9b75-cc6d50c89702 20230524T181718Z geonames append-no-conflicts 4gheap 30260cf | ||
| f91c33d0-ec93-48e1-975e-37476a5c9fe5 20230524T170134Z geonames append-no-conflicts 4gheap 30260cf | ||
| d942b7f9-6506-451d-9dcf-ef502ab3e574 20230524T144827Z geonames append-no-conflicts 4gheap 30260cf | ||
| a33845cc-c2e5-4488-a2db-b0670741ff9b 20230523T213145Z geonames append-no-conflicts | ||
| ``` | ||
|
|
||
| Then, use `compare` to call a `--baseline` test and a `--contender` test for comparison. | ||
|
|
||
| ``` | ||
| opensearch-benchmark compare --baseline=417ed42-6671-9i79-11a1-e367636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7 | ||
| ``` | ||
|
|
||
| You should receive the following response comparing the final benchmark metrics for both tests: | ||
|
|
||
| ``` | ||
| ____ _____ __ ____ __ __ | ||
| / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ | ||
| / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ | ||
| / /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< | ||
| \____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| | ||
| /_/ | ||
| Comparing baseline | ||
| TestExecution ID: 729291a0-ee87-44e5-9b75-cc6d50c89702 | ||
| TestExecution timestamp: 2023-05-24 18:17:18 | ||
| with contender | ||
| TestExecution ID: a33845cc-c2e5-4488-a2db-b0670741ff9b | ||
| TestExecution timestamp: 2023-05-23 21:31:45 | ||
| ------------------------------------------------------ | ||
| _______ __ _____ | ||
| / ____(_)___ ____ _/ / / ___/_________ ________ | ||
| / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \ | ||
| / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/ | ||
| /_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/ | ||
| ------------------------------------------------------ | ||
| Metric Baseline Contender Diff | ||
| -------------------------------------------------------- ---------- ----------- ----------------- | ||
| Min Indexing Throughput [docs/s] 19501 19118 -383.00000 | ||
| Median Indexing Throughput [docs/s] 20232 19927.5 -304.45833 | ||
| Max Indexing Throughput [docs/s] 21172 20849 -323.00000 | ||
| Total indexing time [min] 55.7989 56.335 +0.53603 | ||
| Total merge time [min] 12.9766 13.3115 +0.33495 | ||
| Total refresh time [min] 5.20067 5.20097 +0.00030 | ||
| Total flush time [min] 0.0648667 0.0681833 +0.00332 | ||
| Total merge throttle time [min] 0.796417 0.879267 +0.08285 | ||
| Query latency term (50.0 percentile) [ms] 2.10049 2.15421 +0.05372 | ||
| Query latency term (90.0 percentile) [ms] 2.77537 2.84168 +0.06630 | ||
| Query latency term (100.0 percentile) [ms] 4.52081 5.15368 +0.63287 | ||
| Query latency country_agg (50.0 percentile) [ms] 112.049 110.385 -1.66392 | ||
| Query latency country_agg (90.0 percentile) [ms] 128.426 124.005 -4.42138 | ||
| Query latency country_agg (100.0 percentile) [ms] 155.989 133.797 -22.19185 | ||
| Query latency scroll (50.0 percentile) [ms] 16.1226 14.4974 -1.62519 | ||
| Query latency scroll (90.0 percentile) [ms] 17.2383 15.4079 -1.83043 | ||
| Query latency scroll (100.0 percentile) [ms] 18.8419 18.4241 -0.41784 | ||
| Query latency country_agg_cached (50.0 percentile) [ms] 1.70223 1.64502 -0.05721 | ||
| Query latency country_agg_cached (90.0 percentile) [ms] 2.34819 2.04318 -0.30500 | ||
| Query latency country_agg_cached (100.0 percentile) [ms] 3.42547 2.86814 -0.55732 | ||
| Query latency default (50.0 percentile) [ms] 5.89058 5.83409 -0.05648 | ||
| Query latency default (90.0 percentile) [ms] 6.71282 6.64662 -0.06620 | ||
| Query latency default (100.0 percentile) [ms] 7.65307 7.3701 -0.28297 | ||
| Query latency phrase (50.0 percentile) [ms] 1.82687 1.83193 +0.00506 | ||
| Query latency phrase (90.0 percentile) [ms] 2.63714 2.46286 -0.17428 | ||
| Query latency phrase (100.0 percentile) [ms] 5.39892 4.22367 -1.17525 | ||
| Median CPU usage (index) [%] 668.025 679.15 +11.12499 | ||
| Median CPU usage (stats) [%] 143.75 162.4 +18.64999 | ||
| Median CPU usage (search) [%] 223.1 229.2 +6.10000 | ||
| Total Young Gen GC time [s] 39.447 40.456 +1.00900 | ||
| Total Young Gen GC count 10 11 +1.00000 | ||
| Total Old Gen GC time [s] 7.108 7.703 +0.59500 | ||
| Total Old Gen GC count 10 11 +1.00000 | ||
| Index size [GB] 3.25475 3.25098 -0.00377 | ||
| Total written [GB] 17.8434 18.3143 +0.47083 | ||
| Heap used for segments [MB] 21.7504 21.5901 -0.16037 | ||
| Heap used for doc values [MB] 0.16436 0.13905 -0.02531 | ||
| Heap used for terms [MB] 20.0293 19.9159 -0.11345 | ||
| Heap used for norms [MB] 0.105469 0.0935669 -0.01190 | ||
| Heap used for points [MB] 0.773487 0.772155 -0.00133 | ||
| Heap used for points [MB] 0.677795 0.669426 -0.00837 | ||
| Segment count 136 121 -15.00000 | ||
| Indices Stats(90.0 percentile) [ms] 3.16053 3.21023 +0.04969 | ||
| Indices Stats(99.0 percentile) [ms] 5.29526 3.94132 -1.35393 | ||
| Indices Stats(100.0 percentile) [ms] 5.64971 7.02374 +1.37403 | ||
| Nodes Stats(90.0 percentile) [ms] 3.19611 3.15251 -0.04360 | ||
| Nodes Stats(99.0 percentile) [ms] 4.44111 4.87003 +0.42892 | ||
| Nodes Stats(100.0 percentile) [ms] 5.22527 5.66977 +0.44450 | ||
| ``` | ||
|
|
||
| ## Options | ||
|
|
||
| You can use the following options to customize the results of your test comparison: | ||
|
|
||
| - `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution. | ||
| - `--contender`: The TestExecution ID for the contender being compared to the baseline. | ||
| - `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`. | ||
| - `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. | ||
| - `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. | ||
| - `--show-in-results`: Determines whether or not to include the comparison in the results file. | ||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| --- | ||
| layout: default | ||
| title: download | ||
| nav_order: 60 | ||
| parent: Command reference | ||
| --- | ||
|
|
||
| # download | ||
|
|
||
| Use the `download` command to select which OpenSearch distribution version to download. | ||
|
|
||
| ## Usage | ||
|
|
||
| The following example downloads OpenSearch version 2.7.0: | ||
|
|
||
| ``` | ||
| opensearch-benchmark download --distribution-version=2.7.0 | ||
| ``` | ||
|
|
||
| Benchmark then returns the location of the OpenSearch artifact: | ||
|
|
||
| ``` | ||
| { | ||
| "opensearch": "/Users/.benchmark/benchmarks/distributions/opensearch-2.7.0.tar.gz" | ||
| } | ||
| ``` | ||
|
|
||
| ## Options | ||
|
|
||
| Use the following options to customize how OpenSearch Benchmark downloads OpenSearch: | ||
|
|
||
| - `--provision-config-repository`: Defines the repository from which OpenSearch Benchmark loads `provision-configs` and `provision-config-instances`. | ||
| - `--provision-config-revision`: Defines a specific Git revision in the `provision-config` that OpenSearch Benchmark should use. | ||
| - `--provision-config-path`: Defines the path to the `--provision-config-instance` and any OpenSearch plugin configurations to use. | ||
| - `--distribution-version`: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see [Version history](https://opensearch.org/docs/version-history/). | ||
| - `--distribution-repository`: Defines the repository from where the OpenSearch distribution should be downloaded. Default is `release`. | ||
| - `--provision-config-instance`: Defines the `--provision-config-instance` to use. You can see possible configuration instances using the command `opensearch-benchmark list provision-config-instances`. | ||
| - `--provision-config-instance-params`: A comma-separated list of key-value pairs injected verbatim as variables for the `provision-config-instance`. | ||
| - `--target-os`: The target operating system (OS) for which the OpenSearch artifact should be downloaded. Default is the current OS. | ||
| - `--target-arch`: The name of the CPU architecture for which an artifact should be downloaded. |
Oops, something went wrong.