Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration test against recent OpenSearch versions #757

Conversation

Xtansia
Copy link
Collaborator

@Xtansia Xtansia commented Aug 15, 2024

Description

Describe what this change achieves.

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@Xtansia Xtansia force-pushed the integration-test-new-opensearch-versions branch 18 times, most recently from e84b219 to 059fa22 Compare August 16, 2024 02:14
Signed-off-by: Thomas Farr <[email protected]>
@Xtansia Xtansia force-pushed the integration-test-new-opensearch-versions branch from 059fa22 to 6d6f54a Compare August 16, 2024 02:19
Signed-off-by: Thomas Farr <[email protected]>
@Xtansia Xtansia force-pushed the integration-test-new-opensearch-versions branch from 9f46737 to c62a5d8 Compare August 16, 2024 02:46
@Xtansia Xtansia marked this pull request as ready for review August 16, 2024 02:56
Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what's the root cause of the failures of integration tests?

@Xtansia
Copy link
Collaborator Author

Xtansia commented Aug 17, 2024

So what's the root cause of the failures of integration tests?

The main reason for the sudden seemingly unrelated failure of the segment replication test was due to the low disk watermark causing the replica shards to not get assigned to the second node meaning segrep tasks didn't start. There was calls to wait for green cluster health but they weren't actually validating the response. Digging into it caused me to find a handful of other things that were fragile about the multi-node setup and fixed them along the way. Usage of same data folder for both nodes, not setting seed_hosts and setting of initial_cluster_manager wasn't happening before starting the node.

@Xtansia Xtansia merged commit c8790b7 into opensearch-project:main Aug 17, 2024
49 checks passed
@Xtansia Xtansia deleted the integration-test-new-opensearch-versions branch August 17, 2024 04:39
@Xtansia Xtansia added the backport 1.x Backport to 1.x branch label Aug 17, 2024
@opensearch-trigger-bot
Copy link
Contributor

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-757-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c8790b79b828d0e10b79a65371a9e7e6741a55a3
# Push it to GitHub
git push --set-upstream origin backport/backport-757-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-757-to-1.x.

Xtansia added a commit to Xtansia/opensearch-net that referenced this pull request Aug 18, 2024
…ct#757)

* Integration test against recent OpenSearch versions

Signed-off-by: Thomas Farr <[email protected]>

* Fix multi-node tests

Signed-off-by: Thomas Farr <[email protected]>

* Use debug auditlog

Signed-off-by: Thomas Farr <[email protected]>

---------

Signed-off-by: Thomas Farr <[email protected]>
(cherry picked from commit c8790b7)
dblock pushed a commit that referenced this pull request Aug 19, 2024
* Integration test against recent OpenSearch versions

Signed-off-by: Thomas Farr <[email protected]>

* Fix multi-node tests

Signed-off-by: Thomas Farr <[email protected]>

* Use debug auditlog

Signed-off-by: Thomas Farr <[email protected]>

---------

Signed-off-by: Thomas Farr <[email protected]>
(cherry picked from commit c8790b7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.x Backport to 1.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants