New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add understanding workloads section #6164

Merged

Naarcha-AWS merged 21 commits into main from understanding-workloads

Jan 30, 2024

Collaborator

Naarcha-AWS commented Jan 16, 2024

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Naarcha-AWS added 6 commits

January 16, 2024 11:30


          Add understanding workloads section.

215f6df

Signed-off-by: Naarcha-AWS <[email protected]>


          Add additional anatomy sections

f78251f

Signed-off-by: Naarcha-AWS <[email protected]>


          Add section headers

1c0e724

Signed-off-by: Naarcha-AWS <[email protected]>


          Fix link

782e9e7

Signed-off-by: Naarcha-AWS <[email protected]>


          Fix typos

7289f45

Signed-off-by: Naarcha-AWS <[email protected]>


          Change example to fix build error.

a5043df

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS self-assigned this

Naarcha-AWS added the 3 - Tech review label

Naarcha-AWS marked this pull request as ready for review

January 18, 2024 17:55

Naarcha-AWS requested review from hdhalter, kolchfa-aws, vagimeli, AMoo-Miki, natebower and dlvenable as code owners

January 18, 2024 17:55


          Merge branch 'main' into understanding-workloads

5b8e7d9

Naarcha-AWS added the backport 2.11 label

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated


		## General search clusters

		For benchmarking clusters built for general search use cases, start with [nyc_taxis](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis). The `nyc_taxis` workload data about the rides performed by yellow taxis in New York in 2015.

Contributor

IanHoang Jan 23, 2024

"workload data about" -> "workload contains data about"

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated

+              Consider the following when deciding which workload would work best for benchmarking your cluster:
+              - Consider the use case of your cluster.
+              - Consider what data types your cluster uses by comparing it the data structure of the documents contained in the workload. Each workload contains an example document so you can compare data types. Also, you can go to `index.json` file in the workload to see the data type.

Contributor

IanHoang Jan 23, 2024

"to see the data. type" --> "to see the index mappings and data types"

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated


		## _operations and _test-procedures

		To make the workload more human-readable, operations and test procedures are seperated into two different directors.

Contributor

IanHoang Jan 23, 2024

"different directors" -> "different directories"

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated


		The index.json file contains all of the data mappings and parameters used to index any documents contained inside the workload, as well as the index settings needed when the `create-index` operations in run during the workload.

		For example, in the `nyc_taxis` workload, the `settings` array gives you the ability to customize the number of shards, replicas, and tells the index whether to cache queries or requests. All mappings are based off of a single document, usually called in the `files.txt` file, and includes each mapping parameter and its format, as shown in the following example:

Contributor

IanHoang Jan 23, 2024

Could we rephrase these two paragraphs to something such as:

When OSB creates an index for the workload, it uses the index settings and mappings template found in index.json. Mappings in index.json are based off of the mappings of a single document from the workload's corpus, which can be found in files.txt. For example, the following is the index.json for nyc_taxis workload. Users can customize fields such as number_of_shards, number_of_replicas, query_cache_enabled, and requests_cache_enabled.

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated


		### Schedule

		The `schedule` element contains a list of actions and operations that are run by the workload. Operations run according to the order in which they appear in the `schedule`. The following example illustrates a `schedule` with multiple operations, each defined by its `operation-type`:

Contributor

IanHoang Jan 23, 2024

Maybe we can rephrase this to something like:

The schedule element contains a list of operations that are run according to the order they appear in.

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

IanHoang reviewed

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

IanHoang suggested changes

View reviewed changes

Contributor

IanHoang left a comment

Left a few comments

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS added 4 - Doc review and removed 3 - Tech review labels


          Merge branch 'main' into understanding-workloads

877ac18

Naarcha-AWS mentioned this pull request

Add Running a Workload #6287

Merged

1 task

vagimeli approved these changes

View reviewed changes

Collaborator

vagimeli left a comment

@Naarcha-AWS @natebower Please see my review comments. Thank you, Melissa

_benchmark/user-guide/concepts.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/index.md Outdated Show resolved Hide resolved


          Apply suggestions from code review

ae5ff9d

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Show resolved Hide resolved


          Apply suggestions from code review

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS added 6 - Done but waiting to merge and removed 4 - Doc review labels

natebower reviewed

View reviewed changes

Collaborator

natebower left a comment

@Naarcha-AWS Nice job on this 😄. Please tag me on the rewrites of lines 107 and 161 in the second file so that I can verify them. Thanks!

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/choosing-a-workload.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/index.md Outdated Show resolved Hide resolved

_benchmark/user-guide/understanding-workloads/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS commented

View reviewed changes

_benchmark/user-guide/understanding-workloads/anatomy-of-a-workload.md Outdated Show resolved Hide resolved

Naarcha-AWS and others added 9 commits

January 30, 2024 10:42


          Apply suggestions from code review

f627d11

Co-authored-by: Nathan Bower <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>


          Merge branch 'main' into understanding-workloads


          Apply suggestions from code review

438cb10

Signed-off-by: Naarcha-AWS <[email protected]>


          Update anatomy-of-a-workload.md

d44de4b

Signed-off-by: Naarcha-AWS <[email protected]>


          Fix build errors

243cf0b

Signed-off-by: Naarcha-AWS <[email protected]>


          Update anatomy-of-a-workload.md

3edb6d4

Signed-off-by: Naarcha-AWS <[email protected]>


          Update anatomy-of-a-workload.md

2def131

Signed-off-by: Naarcha-AWS <[email protected]>


          Update concepts.md

f66089a

Signed-off-by: Naarcha-AWS <[email protected]>


          Update index.md

dc9d495

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS merged commit bf4ae72 into main

4 checks passed

Naarcha-AWS deleted the understanding-workloads branch

January 30, 2024 17:50

opensearch-trigger-bot bot pushed a commit that referenced this pull request


          Add understanding workloads section (#6164)

f82219a

* Add understanding workloads section.

Signed-off-by: Naarcha-AWS <[email protected]>

* Add additional anatomy sections

Signed-off-by: Naarcha-AWS <[email protected]>

* Add section headers

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix link

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix typos

Signed-off-by: Naarcha-AWS <[email protected]>

* Change example to fix build error.

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update anatomy-of-a-workload.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix build errors

Signed-off-by: Naarcha-AWS <[email protected]>

* Update anatomy-of-a-workload.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update anatomy-of-a-workload.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update concepts.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update index.md

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
(cherry picked from commit bf4ae72)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot bot mentioned this pull request

[Backport 2.11] Add understanding workloads section #6298

Merged

Naarcha-AWS pushed a commit that referenced this pull request


          Add understanding workloads section (#6164) (#6298)

7c0785a

* Add understanding workloads section.



* Add additional anatomy sections



* Add section headers



* Fix link



* Fix typos



* Change example to fix build error.



* Apply suggestions from code review



* Apply suggestions from code review



* Apply suggestions from code review




* Apply suggestions from code review




* Apply suggestions from code review





* Apply suggestions from code review



* Update anatomy-of-a-workload.md



* Fix build errors



* Update anatomy-of-a-workload.md



* Update anatomy-of-a-workload.md



* Update concepts.md



* Update index.md



---------





(cherry picked from commit bf4ae72)

Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

hdhalter added 3 - Done and removed 6 - Done but waiting to merge labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

vagimeli vagimeli approved these changes

natebower natebower left review comments

IanHoang IanHoang requested changes

hdhalter Awaiting requested review from hdhalter

kolchfa-aws Awaiting requested review from kolchfa-aws kolchfa-aws is a code owner

AMoo-Miki Awaiting requested review from AMoo-Miki AMoo-Miki is a code owner

dlvenable Awaiting requested review from dlvenable dlvenable is a code owner

Labels

3 - Done backport 2.11