Experiment 1 and 2: Scale Testing OSB with Distributed Workload Generation

[IanHoang]

Background

We're aware that OSB has limitations, but we don't know exactly under what circumstances they occur. For example, it's known that OSB can test single node and small multi node (3,4,5 etc.) OpenSearch clusters. However, when OSB tests a really large cluster (unclear on how large but let's just say 20 nodes), it tends to fail. To combat this, OSB has a feature called Distributed Workload Generation (DWG) which basically allows users to get over that hurdle of testing large 20 node clusters.

However, DWG has only been used to a certain extent and we do not know its limitations. Some questions we want to answer for this feature are:

• How many nodes can we use for DWG? Is it only max 3 nodes?
• What parameters do we need to ensure that users include when using DWG?
• Does running DWG on a single node cluster or small multi node clusters perform better than just using standard OSB (1 node) ?
  These aren't the only questions but these have been unanswered for some time.

See RFC #416 for more details.

Strategy

Experiment 1: Comparing OSB Without DWG and OSB With DWG

We should perform a series of tests for standard OSB And DWG of various clusters. When testing OSB without DWG and with DWG, we should run the following tests.

• workloads: nyc_taxis
• Run against: single node cluster, 3M3D multi-node cluster, 3M15D multi-node cluster
• For each cluster, run the workload ~5 times (to collect consistent data)

In other words, we should run 30 tests total, 15 tests for each series.

Experiment 2: How many nodes can OSB use with DWG

• workloads: nyc_taxis
• Run against: 3M3D multi-node cluster
• Run a few rounds, one with 3 LG Hosts, then 5 LG Hosts, and then try 10 LG Hosts

If possible, since we're running this with a search workload, we should also run this with a log analytics workload such as http_logs.

When assessing the results, we should aim to see if there are performance differences and when they occur.

[IanHoang]

Newer and improved issue here: #506