Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky QueryProfilePhaseTests.testTerminateAfterEarlyTermination #11683

Conversation

ticheng-aws
Copy link
Contributor

@ticheng-aws ticheng-aws commented Dec 28, 2023

Description

QueryProfilePhaseTests.testTerminateAfterEarlyTermination is flaky not only in the concurrent search case but also in the non-concurrent search case. In cases where the test query returns some total hits, it is possible that the score breakdown timing for certain rewritten or sub-queries is still zero. Consequently, we need to update the assertion check accordingly.
Additionally, in the concurrent search case, there is a particular scenario where some slices have zero score breakdown timing, while others have non-zero score breakdown timing. In this case, the min_score stat result will be expected zero. I ran the new changes 1000+ times to make sure that they all passed.

Related Issues

Resolves #11611

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added bug Something isn't working flaky-test Random test failure that succeeds on second run Search:Query Insights labels Dec 28, 2023
@ticheng-aws ticheng-aws added skip-changelog and removed bug Something isn't working flaky-test Random test failure that succeeds on second run Search:Query Insights labels Dec 28, 2023
Copy link
Contributor

github-actions bot commented Dec 28, 2023

Compatibility status:

Checks if related components are compatible with change 3c00409

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

@github-actions github-actions bot added bug Something isn't working flaky-test Random test failure that succeeds on second run Search:Query Insights labels Dec 28, 2023
Copy link
Contributor

❕ Gradle check result for 88eb13d: UNSTABLE

  • TEST FAILURES:
      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Dec 28, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (316b60a) 71.35% compared to head (3c00409) 71.48%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #11683      +/-   ##
============================================
+ Coverage     71.35%   71.48%   +0.12%     
- Complexity    59201    59283      +82     
============================================
  Files          4906     4906              
  Lines        278198   278198              
  Branches      40422    40422              
============================================
+ Hits         198520   198865     +345     
+ Misses        63229    62834     -395     
- Partials      16449    16499      +50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ticheng-aws ticheng-aws force-pushed the cs-testTerminateAfterEarlyTermination branch from 88eb13d to 3c00409 Compare December 28, 2023 01:12
Copy link
Contributor

❕ Gradle check result for 3c00409: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ticheng-aws
Copy link
Contributor Author

❕ Gradle check result for 3c00409: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Known issue #9191

@sohami sohami added the backport 2.x Backport to 2.x branch label Jan 3, 2024
@sohami sohami merged commit bb3959d into opensearch-project:main Jan 3, 2024
114 of 117 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 3, 2024
Signed-off-by: Ticheng Lin <[email protected]>
(cherry picked from commit bb3959d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sohami pushed a commit that referenced this pull request Jan 3, 2024
(cherry picked from commit bb3959d)

Signed-off-by: Ticheng Lin <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch bug Something isn't working flaky-test Random test failure that succeeds on second run Search:Query Insights skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] QueryProfilePhaseTests.testTerminateAfterEarlyTermination is flaky
2 participants