[BUG] Possible performance regression from "pmc" workload on 2.12.0 for aggregation based queries

[arjunrajnetapp]

Describe the bug

We have observed a noticeable regression for aggregation based queries when running the OpenSearch Benchmark tool against 2.12.0 using the "pmc" workload. Benchmarking has shown a >400% decrease in the performance of the articles_monthly_agg_uncached task compared to 2.11.1.

Related component

Search:Aggregations

To Reproduce

1. Set up the following OpenSearch clusters:
   1. 3 node OpenSearch cluster with 2.12.0, dedicated manager disabled, security plugin enabled
   2. 3 node OpenSearch cluster with 2.11.1, dedicated manager disabled, security plugin enabled
2. Set up a stressor box with the benchmark tool using the the same node size and is located in the same region and availability zone as nodes targeted for benchmarking to minimise network variance.
3. Run the the OpenSearch Benchmark tool with the pmc and with unlimited throughput configured through workload_params.json -

{
"target_throughput": 0
}

4. Compare the performance of the 50th percentile latency metric for the articles_monthly_agg_uncached task.

Expected behavior

Performance improvements for the articles_monthly_agg_uncached task in line with previous report.

Additional Details

Plugins
Please list all plugins currently enabled.
Security Plugin

Screenshots
If applicable, add screenshots to help explain your problem.
N/A

Host/Environment (please complete the following information):

• OS: Debian 11.8
• 3 node OpenSearch Cluster
• Dedicated manager is disabled
• Versions compared: 2.12.0 vs 2.11.1
• Node sizes tested:
  • AWS - m6g.large - 120 GiB (GP3 EBS SSD) - 8 GiB RAM
  • AWS - r6g.xlarge - 400 GiB (GP3 EBS SSD) - 32 GiB RAM

Additional context
Add any other context about the problem here.

We did verify that there are indeed improvements to aggregation based queries from the http_logs workload as previously reported, however for the pmc workload we have observed that the articles_monthly_agg_uncached task shows a noticeable regression of upwards of 400% decrease in performance.

The benchmarking results and comparison are attached in the following spreadsheet -
2.11.1 vs 2.12.0 pmc results comparison.xlsx

[harshavamsi]

Thanks @arjunrajnetapp for filing this issue.

@bowenlan-amzn @jainankitk can we benchmark this on PMC and see what the numbers look like?

[jainankitk]

@bowenlan-amzn - Can you help verify this?

[bowenlan-amzn]

I'm trying to reproduce this.

[peternied]

[Triage - attendees 1 2 3 4 5 6]
@arjunrajnetapp Thanks for creating this issue and thanks @bowenlan-amzn for digging in on it

[bowenlan-amzn]

Fix/Enhancement has merged in main and backporting to 2.x and 2.14

[bowenlan-amzn]

Can see from our nightly benchmark