[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #5746

dreamer-89 · 2023-01-07T18:47:24Z

Description

This change is a clean up and improvement during peer recovery with segment replication. The change was originally introduced in #5332. This change performs

Untie IndicesClusterStateService from SegmentReplicationTargetService
Fail fast during recovery.
Updates SegmentReplicationSourceHandlerTests to use correct engine factory and updates existing unit test classes to consider replica recovery
Adds one integration test testNewlyAddedReplicaIsUpdated which verifies happy path scenario. The change moves this test along with testAddNewReplicaFailure to SegmentReplicationRelocationIT which is more natural class for these ITs. Both tests are muted as they tend to fail on CI. These relocation test failures are tracked in [BUG] failing IT test : SegmentReplicationRelocationIT #6065
The change fixes some unit test assertions, identified with this change.

Issues Resolved

#5743

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-07T19:08:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9006/
CommitID: 0d133e6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-08T02:51:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/9006/

CommitID: 0d133e6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

There was a 2.x version bump to 2.6.0 from 2.5.0 in 06c1712. This also needs change in main.

[Edit]: This is already done in #5745. This PR needs rebase against main.

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.5.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.x/distribution/archives/linux-tar/build/distributions/opensearch-min-2.5.0-SNAPSHOT-linux-x64.tar.gz

dreamer-89 · 2023-01-09T20:05:58Z

As this PR change recovery flow for replica, it also needed changes in unit tests, which were failing in last gradle check. There are still 3 legitimate unit test failures related to SegmentReplicationIndexShardTests which needs more dive. These unit tests are failing during doc count assertions. The target's translog contains all documents rather than ones ingested post round of segment replication.

assertEquals(additonalDocs, nextPrimary.translogStats().estimatedNumberOfOperations());

github-actions · 2023-01-09T21:38:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9104/
CommitID: 300f141
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-09T22:01:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9105/
CommitID: 60ca0ac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-28T23:24:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10116/
CommitID: 00ee972
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-01T01:59:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10364/
CommitID: c1837d6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

mch2 · 2023-02-01T07:18:18Z

test/framework/src/main/java/org/opensearch/index/shard/IndexShardTestCase.java

@@ -847,7 +847,7 @@ protected DiscoveryNode getFakeDiscoNode(String id) {
    }

    protected void recoverReplica(IndexShard replica, IndexShard primary, boolean startReplica) throws IOException {
-        recoverReplica(replica, primary, startReplica, (a) -> null);
+        recoverReplica(replica, primary, startReplica, getReplicationFunc(replica));


love this add to our tests!

...ernalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationRelocationIT.java

mch2 · 2023-02-01T07:26:10Z

...ernalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationRelocationIT.java

+            .actionGet();
+        assertTrue(clusterHealthResponse.isTimedOut());
+        ensureYellow(INDEX_NAME);
+        IndicesService indicesService = internalCluster().getInstance(IndicesService.class, replicaNode);


I wonder if the flakiness in these tests is asserting that IndicesService exists. After the round of SR fails the shard will fail and node will be yellow, it will then try and spin up and recover the shard again, causing this assertion to trip?

to confirm you could maybe flip this assertion to true and wrap it in an assertBusy and see that it always succeeds.

I wonder if the flakiness in these tests is asserting that IndicesService exists.

Yes, it was flaky and fails asserting index on replica doesn't exist.

After the round of SR fails the shard will fail and node will be yellow, it will then try and spin up and recover the shard again, causing this assertion to trip?

Bingo, yes you are right. The recovery kicks in again post failure. Modified the test to wait for first round of recovery before assertion.

… sync from source Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>

github-actions · 2023-02-01T20:57:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10399/
CommitID: a19dccc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Suraj Singh <[email protected]>

github-actions · 2023-02-01T23:39:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10441/
CommitID: 9a03af0

mch2

LGTM thanks!

opensearch-trigger-bot · 2023-02-01T23:54:12Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5746-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 53d54de90423e31af7ca2514b953d77df4ac2be4
# Push it to GitHub
git push --set-upstream origin backport/backport-5746-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5746-to-2.x.

… sync from peer recovery source (opensearch-project#5746) * [Segment Replication] For replica recovery, force segment replication sync from source Signed-off-by: Suraj Singh <[email protected]> * Rebase against main Signed-off-by: Suraj Singh <[email protected]> * Fix unit test Signed-off-by: Suraj Singh <[email protected]> * PR feedback Signed-off-by: Suraj Singh <[email protected]> * Fix remote store recovery test Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>

… sync from peer recovery source (#5746) (#6149) * [Segment Replication] For replica recovery, force segment replication sync from source * Rebase against main * Fix unit test * PR feedback * Fix remote store recovery test --------- Signed-off-by: Suraj Singh <[email protected]>

… sync from peer recovery source (opensearch-project#5746) * [Segment Replication] For replica recovery, force segment replication sync from source Signed-off-by: Suraj Singh <[email protected]> * Rebase against main Signed-off-by: Suraj Singh <[email protected]> * Fix unit test Signed-off-by: Suraj Singh <[email protected]> * PR feedback Signed-off-by: Suraj Singh <[email protected]> * Fix remote store recovery test Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>

dreamer-89 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, tlfeng, VachaShah and xuezhou25 as code owners January 7, 2023 18:47

dreamer-89 added the skip-changelog label Jan 7, 2023

dreamer-89 force-pushed the update_replica_recovery_segrep branch from 0d133e6 to ce607cb Compare January 8, 2023 02:53

This comment was marked as outdated.

Sign in to view

dreamer-89 force-pushed the update_replica_recovery_segrep branch 2 times, most recently from 2d7ff7b to 300f141 Compare January 9, 2023 20:00

This comment was marked as outdated.

Sign in to view

dreamer-89 force-pushed the update_replica_recovery_segrep branch from 300f141 to 60ca0ac Compare January 9, 2023 20:10

dreamer-89 changed the title ~~[Segment Replication] Force segment replication sync from peer recovery source~~ [Segment Replication] For replica recovery, force segment replication sync from peer recovery source Jan 28, 2023

dreamer-89 force-pushed the update_replica_recovery_segrep branch from 60ca0ac to 00ee972 Compare January 28, 2023 22:57

dreamer-89 force-pushed the update_replica_recovery_segrep branch from 00ee972 to c1837d6 Compare February 1, 2023 01:34

mch2 reviewed Feb 1, 2023

View reviewed changes

dreamer-89 added 4 commits February 1, 2023 09:51

[Segment Replication] For replica recovery, force segment replication…

b46d307

… sync from source Signed-off-by: Suraj Singh <[email protected]>

Rebase against main

4e52d0a

Signed-off-by: Suraj Singh <[email protected]>

Fix unit test

e4e107d

Signed-off-by: Suraj Singh <[email protected]>

PR feedback

a19dccc

Signed-off-by: Suraj Singh <[email protected]>

dreamer-89 force-pushed the update_replica_recovery_segrep branch from c1837d6 to a19dccc Compare February 1, 2023 18:55

Fix remote store recovery test

9a03af0

Signed-off-by: Suraj Singh <[email protected]>

mch2 approved these changes Feb 1, 2023

View reviewed changes

dreamer-89 merged commit 53d54de into opensearch-project:main Feb 1, 2023

dreamer-89 added the backport 2.x Backport to 2.x branch label Feb 1, 2023

This was referenced Feb 2, 2023

[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #6149

Merged

[Segment Replication] Update replica recovery to perform replication from source #5743

Closed

dreamer-89 mentioned this pull request Mar 10, 2023

[Segment Replication] Prevent segment replication for ineligible targets during recovery #6611

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #5746

[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #5746

dreamer-89 commented Jan 7, 2023 •

edited

Loading

github-actions bot commented Jan 7, 2023

dreamer-89 commented Jan 8, 2023 •

edited

Loading

Gradle Check (Jenkins) Run Completed with:

This comment was marked as outdated.

dreamer-89 commented Jan 9, 2023

This comment was marked as outdated.

github-actions bot commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

github-actions bot commented Jan 28, 2023

github-actions bot commented Feb 1, 2023

mch2 Feb 1, 2023

mch2 Feb 1, 2023

dreamer-89 Feb 1, 2023

github-actions bot commented Feb 1, 2023

github-actions bot commented Feb 1, 2023

mch2 left a comment

opensearch-trigger-bot bot commented Feb 1, 2023

[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #5746

[Segment Replication] For replica recovery, force segment replication sync from peer recovery source #5746

Conversation

dreamer-89 commented Jan 7, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 7, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 8, 2023 • edited Loading

Gradle Check (Jenkins) Run Completed with:

This comment was marked as outdated.

dreamer-89 commented Jan 9, 2023

This comment was marked as outdated.

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 Feb 1, 2023

Choose a reason for hiding this comment

mch2 Feb 1, 2023

Choose a reason for hiding this comment

dreamer-89 Feb 1, 2023

Choose a reason for hiding this comment

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Feb 1, 2023

dreamer-89 commented Jan 7, 2023 •

edited

Loading

dreamer-89 commented Jan 8, 2023 •

edited

Loading