Correctly calculate doc count error at the slice level for concurrent segment search

[jed326]

Description

This PR fixes how doc count error is calculated for terms aggregations for conurrent segment search and sets the slice_size heuristic introduced in #11585 to be equal to the shard_size.

The crux of this change is the introduction of hasSliceLevelDocCountError, to indicate to the coordinator whether or not the slice_size is the reason for why the error exists. This affects how the top level aggregation error is then calculated in the single shard (and single slice) scenarios.

Related Issues

Resolves #11680
Resolves #11702

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 1f31849: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4b4aacd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jed326]

@sohami @reta could you please help review? Thanks!

[github-actions]

❌ Gradle check result for 7e64bf8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jed326]

@sohami @reta I updated the PR and also added some write ups related to the big issues we're seeing here. Both are referenced by code comments as well. See:

• [Concurrent Segment Search] shard_min_doc_count is not enforced at the slice level #11847
• [Concurrent Segment Search] Doc count error needs to be computed at the slice level #11680

Taking a small step back, I do think the way that these terms aggregators are currently written is quite fragile and each piece of information seems tacked on to the previous. doc_count_error in particular looks like it was implemented as an afterthought and I do think there is a lot of room to simplify the code here. I will try to open a separate issue with some of those thoughts later.

[github-actions]

✅ Gradle check result for 662ab41: SUCCESS

[jed326]

Created a new docs issue to add a lot of the details discussed here to the opensearch docs. The terms and sig terms docs are very lacking currently.

• [DOC] Terms and Significant Terms aggs docs are missing several search parameters and explanations documentation-website#6129

[github-actions]

❌ Gradle check result for 7571351: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jed326]

❌ Gradle check result for 7571351: FAILURE

org.opensearch.search.aggregations.bucket.TermsDocCountErrorIT.testDoubleValueFieldSingleShard {p0={"search.concurrent_segment_search.enabled":"true"}}
org.opensearch.search.aggregations.bucket.TermsDocCountErrorIT.testStringValueFieldSingleShard {p0={"search.concurrent_segment_search.enabled":"true"}}
org.opensearch.search.aggregations.bucket.TermsDocCountErrorIT.testLongValueFieldSingleShard {p0={"search.concurrent_segment_search.enabled":"true"}}

Looks like there are a few more tests I need to fix actually. I'm going to just force merge all of the shards in this test case to 1 segment since the purpose of this test class is to run assertions on the doc count error and we know that there will be a difference in doc count error when we use concurrent search. Will add a comment explaining such.

[github-actions]

❌ Gradle check result for 71a7472: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jed326]

https://build.ci.opensearch.org/job/gradle-check/32022/

• [BUG] Flaky Test - org.opensearch.index.shard.IndexShardTests.testCommitLevelRestoreShardFromRemoteStore  #10704

[github-actions]

❕ Gradle check result for 71a7472: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.