Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenSearch crashes on closed client connection before search reply when total ops higher compared to expected #4143

Merged
merged 1 commit into from
Aug 5, 2022

Conversation

reta
Copy link
Collaborator

@reta reta commented Aug 5, 2022

Signed-off-by: Andriy Redko [email protected]

Description

In some circumstances, when running a search operation, if the client TCP connection is closed before the search operation is completed, opensearch crashes with:

[2022-07-20T17:51:36,145][ERROR][org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler] fatal error in thread [opensearch[mycluster-elasticsearch-3][search][T#8]], exiting
java.lang.AssertionError: unexpected higher total ops [15] compared to expected [14]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:465) ~[opensearch-1.3.4.jar:1.3.4]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$2(AbstractSearchAsyncAction.java:303) ~[opensearch-1.3.4.jar:1.3.4]

Issues Resolved

Closes #3962

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…en total ops higher compared to expected

Signed-off-by: Andriy Redko <[email protected]>
@reta reta requested a review from a team as a code owner August 5, 2022 17:21
@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2022

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented Aug 5, 2022

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.64%. Comparing base (fbe93d4) to head (addd0a8).
Report is 2980 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4143      +/-   ##
============================================
+ Coverage     70.55%   70.64%   +0.08%     
- Complexity    57013    57093      +80     
============================================
  Files          4603     4603              
  Lines        274478   274482       +4     
  Branches      40201    40201              
============================================
+ Hits         193658   193905     +247     
+ Misses        64511    64336     -175     
+ Partials      16309    16241      -68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andrross andrross merged commit 203f44e into opensearch-project:main Aug 5, 2022
@andrross andrross added the backport 2.x Backport to 2.x branch label Aug 5, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143)

Signed-off-by: Andriy Redko <[email protected]>
(cherry picked from commit 203f44e)
@dbonf
Copy link

dbonf commented Aug 5, 2022

will this be backported to 1.3 please?

@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

will this be backported to 1.3 please?

Yes, on it

reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

@dbonf would you be able to give it a try on 1.3.5-SNAPSHOT? thank you!

@dbonf
Copy link

dbonf commented Aug 5, 2022

@reta sure, do I need to compile it or is it already available somewhere?

reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

@reta sure, do I need to compile it or is it already available somewhere?

Good question, I think if you could compile (once #4148 is in), would be awesome, I am not 100% sure we publish snapshots

dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4144)

Signed-off-by: Andriy Redko <[email protected]>
(cherry picked from commit 203f44e)

Co-authored-by: Andriy Redko <[email protected]>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4145)

Signed-off-by: Andriy Redko <[email protected]>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4146)

Signed-off-by: Andriy Redko <[email protected]>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4147)

Signed-off-by: Andriy Redko <[email protected]>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4148)

Signed-off-by: Andriy Redko <[email protected]>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
…en total ops higher compared to expected (#4143) (#4149)

Signed-off-by: Andriy Redko <[email protected]>
@dbonf
Copy link

dbonf commented Aug 6, 2022

@reta I managed to build from reta:backport/backport-4143-to-1.3 and the tests are promising, no crashes

dreamer-89 pushed a commit to dreamer-89/OpenSearch that referenced this pull request Aug 12, 2022
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <[email protected]>
@dreamer-89 dreamer-89 added the v2.3.0 'Issues and PRs related to version v2.3.0' label Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch v2.3.0 'Issues and PRs related to version v2.3.0'
Projects
None yet
5 participants