stopping replication before clean up of indices

[sricharanvuppu]

Signed-off-by: sricharanvuppu [email protected]

Description

stopping replication before clean up of indices

Issues Resolved

#412

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #619 (e63e832) into main (f3fee90) will increase coverage by 2.34%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #619      +/-   ##
============================================
+ Coverage     70.76%   73.10%   +2.34%     
- Complexity      966      983      +17     
============================================
  Files           141      141              
  Lines          4600     4600              
  Branches        507      507              
============================================
+ Hits           3255     3363     +108     
+ Misses         1013      923      -90     
+ Partials        332      314      -18     

Impacted Files	Coverage Δ
...rch/replication/task/shard/ShardReplicationTask.kt	75.18% <0.00%> (+1.45%)	⬆️
...ication/metadata/state/ReplicationStateMetadata.kt	51.72% <0.00%> (+1.72%)	⬆️
...ation/task/shard/ShardReplicationChangesTracker.kt	63.63% <0.00%> (+3.03%)	⬆️
...ication/action/status/ReplicationStatusResponse.kt	50.90% <0.00%> (+3.63%)	⬆️
...lication/action/status/TranportShardsInfoAction.kt	96.66% <0.00%> (+16.66%)	⬆️
.../action/status/TransportReplicationStatusAction.kt	97.82% <0.00%> (+18.47%)	⬆️
...cation/action/changes/TransportGetChangesAction.kt	84.12% <0.00%> (+20.63%)	⬆️
...rch/replication/action/status/ShardInfoResponse.kt	91.59% <0.00%> (+56.30%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[ankitkala]

@saikaranam-amazon Just a thought, should we also handle the exception cases(if stop called failed) here maybe?
e.g.

• Ensure that all replication related tasks are cancelled.
• Cleaning up replication metadata as well, if possible.

[saikaranam-amazon]

yes, we can target for replication tasks but
any failure to remove the blocks are still applicable in this case (as we currently, we don't have clean way to do that) and with metadata present, we might attempt to clean the state eventually.

Probably, we should prioritise the issue to cleanly exit _stop replication API call. Thoughts?

[ankitkala]

Makes sense. In that case, at least we need to randomize the leader and follower index names in all tests to avoid any cascading failures incase we aren't able to do a clean _stop. Probably we can fast follow after this PR.

[ankitkala]

From Follower stats API response, do we need to filter any indices? I mean just wanted to confirm that we are iterating over syncing/bootstrapping/failed indices but not stopped ones.

[sricharanvuppu]

follower indices whose status is REPLICATION NOT IN PROGRESS are giving 200 OK response when passed to stopReplication API, If those indices are skipped then DELETE request is failing

[saikaranam-amazon]

Can we remove the _stop API calls from the previous tests?
I think, that should be sufficient.

[saikaranam-amazon]

yes, we can target for replication tasks but
any failure to remove the blocks are still applicable in this case (as we currently, we don't have clean way to do that) and with metadata present, we might attempt to clean the state eventually.

Probably, we should prioritise the issue to cleanly exit _stop replication API call. Thoughts?

[saikaranam-amazon]

Lets remove the calls from other integration tests as well - StopIT, StartIT etc?

[sricharanvuppu]

ok

[ankitkala]

Can you fix the DCO check?

[opensearch-trigger-bot]

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.3 1.3
# Navigate to the new working tree
cd .worktrees/backport-1.3
# Create a new branch
git switch --create backport/backport-619-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-1.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-619-to-1.3.

[opensearch-trigger-bot]

The backport to 1.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.1 1.1
# Navigate to the new working tree
cd .worktrees/backport-1.1
# Create a new branch
git switch --create backport/backport-619-to-1.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-1.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.1

Then, create a pull request where the base branch is 1.1 and the compare/head branch is backport/backport-619-to-1.1.

[opensearch-trigger-bot]

The backport to 2.0 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.0 2.0
# Navigate to the new working tree
cd .worktrees/backport-2.0
# Create a new branch
git switch --create backport/backport-619-to-2.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.0

Then, create a pull request where the base branch is 2.0 and the compare/head branch is backport/backport-619-to-2.0.

[opensearch-trigger-bot]

The backport to 1.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.2 1.2
# Navigate to the new working tree
cd .worktrees/backport-1.2
# Create a new branch
git switch --create backport/backport-619-to-1.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-1.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.2

Then, create a pull request where the base branch is 1.2 and the compare/head branch is backport/backport-619-to-1.2.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-619-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-619-to-1.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-619-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-619-to-2.x.

[opensearch-trigger-bot]

The backport to 2.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.3 2.3
# Navigate to the new working tree
cd .worktrees/backport-2.3
# Create a new branch
git switch --create backport/backport-619-to-2.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.3

Then, create a pull request where the base branch is 2.3 and the compare/head branch is backport/backport-619-to-2.3.

[opensearch-trigger-bot]

The backport to 2.4 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.4 2.4
# Navigate to the new working tree
cd .worktrees/backport-2.4
# Create a new branch
git switch --create backport/backport-619-to-2.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.4
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.4

Then, create a pull request where the base branch is 2.4 and the compare/head branch is backport/backport-619-to-2.4.

[opensearch-trigger-bot]

The backport to 2.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.1 2.1
# Navigate to the new working tree
cd .worktrees/backport-2.1
# Create a new branch
git switch --create backport/backport-619-to-2.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.1

Then, create a pull request where the base branch is 2.1 and the compare/head branch is backport/backport-619-to-2.1.

[opensearch-trigger-bot]

The backport to 2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.2 2.2
# Navigate to the new working tree
cd .worktrees/backport-2.2
# Create a new branch
git switch --create backport/backport-619-to-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad8e8ded88c2548a6811855929d5c71f5cea4e3e
# Push it to GitHub
git push --set-upstream origin backport/backport-619-to-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.2

Then, create a pull request where the base branch is 2.2 and the compare/head branch is backport/backport-619-to-2.2.