[DOCUMENTATION] Performance Testing Best Practices

[IanHoang]

Is your feature request related to a problem? Please describe

Many users ask us how to recreate the nightly runs or best practices related to performance testing.

Describe the solution you'd like

We should have an article detailing high-level points.

Describe alternatives you've considered

No response

Additional context

No response

[IanHoang]

@rishabh6788 We can put something together for @Naarcha-AWS based your experience with constructing the nightly runs.

[IanHoang]

In addition to performance testing best practices, we should create documentation for explaining the public nightly runs and dashboards. Users have expressed that the docs and content in dashboards is not intuitive or self explanatory.

Some points we should address in this document:

• unit to the numbers that are in the graphs? it is milliseconds? requests per second?
• How is the metrics calculated? important so customers can compare against other competitors they are looking at.
• Why each metric is important and how it translates to real world application?
• Maybe a video on the OpenSearch youtube channel with a walkthrough of the metrics would be super helpful and can also be embedded on the webpage.

This documentation should be straightforward so that readers of all backgrounds can understand.

[IanHoang]

• Outline of headers / areas to focus on
  • What kinds of setup to use (cluster settings, index settings, lg host machine types,)
  • Familiarity with your workload and cluster (example: understanding queries that your workload experiences)
  • Basics on Key Metrics (Service Time, Latency, Took Time, QPS) to look at
  • Common Gotchas (leverage what users are facing)
    • Address unrealistic expectations of throughput testing