Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9629

mch2 · 2023-08-30T01:09:13Z

Description

This change prevents snapshots from being taken with segment replication during shard promotion. This prevents snapshots from being taken on stale data.

Related Issues

Resolves #9626

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java

github-actions · 2023-08-30T01:42:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23775/
CommitID: 94a1812
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/snapshots/SegmentReplicationSnapshotIT.java

github-actions · 2023-08-30T01:52:21Z

Compatibility status:

Checks if related components are compatible with change 94a1812

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T07:55:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23932/
CommitID: 94a1812
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-02T21:52:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24248/
CommitID: af301fb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-02T22:13:00Z

Compatibility status:

Checks if related components are compatible with change af301fb

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-02T22:20:34Z

Compatibility status:

Checks if related components are compatible with change d726ff3

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-02T22:28:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24249/
CommitID: d726ff3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…hard has not completed failover. Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

github-actions · 2023-09-02T23:37:31Z

Compatibility status:

Checks if related components are compatible with change 8a76b7e

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-02T23:53:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseAllShards
      1 org.opensearch.client.PitIT.testDeleteAllAndListAllPits

URL: https://build.ci.opensearch.org/job/gradle-check/24254/
CommitID: 8a76b7e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-02T23:54:19Z

Codecov Report

Merging #9629 (8a76b7e) into main (e98ded6) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #9629      +/-   ##
============================================
+ Coverage     71.07%   71.16%   +0.09%     
- Complexity    57730    57795      +65     
============================================
  Files          4806     4806              
  Lines        272238   272240       +2     
  Branches      39729    39730       +1     
============================================
+ Hits         193480   193737     +257     
+ Misses        62531    62250     -281     
- Partials      16227    16253      +26

Files Changed	Coverage Δ
...rg/opensearch/snapshots/SnapshotShardsService.java	`69.49% <100.00%> (+4.51%)`	⬆️

... and 466 files with indirect coverage changes

dreamer-89

Thanks @mch2 for fixing this!

…hard has not completed failover. (#9629) * Segment Replication - Block snapshot creation if the target primary shard has not completed failover. Signed-off-by: Marc Handalian <[email protected]> * Add unit test and remove non deterministic IT. Signed-off-by: Marc Handalian <[email protected]> * spotless Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit 731e043) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…hard has not completed failover. (#9629) (#9721) * Segment Replication - Block snapshot creation if the target primary shard has not completed failover. * Add unit test and remove non deterministic IT. * spotless --------- (cherry picked from commit 731e043) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…hard has not completed failover. (opensearch-project#9629) * Segment Replication - Block snapshot creation if the target primary shard has not completed failover. Signed-off-by: Marc Handalian <[email protected]> * Add unit test and remove non deterministic IT. Signed-off-by: Marc Handalian <[email protected]> * spotless Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…hard has not completed failover. (opensearch-project#9629) * Segment Replication - Block snapshot creation if the target primary shard has not completed failover. Signed-off-by: Marc Handalian <[email protected]> * Add unit test and remove non deterministic IT. Signed-off-by: Marc Handalian <[email protected]> * spotless Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…hard has not completed failover. (opensearch-project#9629) * Segment Replication - Block snapshot creation if the target primary shard has not completed failover. Signed-off-by: Marc Handalian <[email protected]> * Add unit test and remove non deterministic IT. Signed-off-by: Marc Handalian <[email protected]> * spotless Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

dreamer-89 reviewed Aug 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java Show resolved Hide resolved

dreamer-89 reviewed Aug 30, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/snapshots/SegmentReplicationSnapshotIT.java Outdated Show resolved Hide resolved

mch2 added 3 commits September 2, 2023 15:54

Segment Replication - Block snapshot creation if the target primary s…

45b7348

…hard has not completed failover. Signed-off-by: Marc Handalian <[email protected]>

Add unit test and remove non deterministic IT.

e430de3

Signed-off-by: Marc Handalian <[email protected]>

spotless

8a76b7e

Signed-off-by: Marc Handalian <[email protected]>

mch2 force-pushed the snapshotfix branch from d726ff3 to 8a76b7e Compare September 2, 2023 23:06

mch2 added the skip-changelog label Sep 2, 2023

dreamer-89 approved these changes Sep 4, 2023

View reviewed changes

dreamer-89 merged commit 731e043 into opensearch-project:main Sep 4, 2023
14 of 15 checks passed

dreamer-89 added the backport 2.x Backport to 2.x branch label Sep 4, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 4, 2023

[Backport 2.x] Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9629

Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9629

mch2 commented Aug 30, 2023 •

edited

Loading

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Sep 2, 2023

github-actions bot commented Sep 2, 2023

github-actions bot commented Sep 2, 2023

github-actions bot commented Sep 2, 2023

github-actions bot commented Sep 2, 2023

github-actions bot commented Sep 2, 2023

codecov bot commented Sep 2, 2023

dreamer-89 left a comment

Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9629

Segment Replication - Block snapshot creation if the target primary shard has not completed failover. #9629

Conversation

mch2 commented Aug 30, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 2, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 2, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 2, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 2, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 2, 2023

Codecov Report

dreamer-89 left a comment

Choose a reason for hiding this comment

mch2 commented Aug 30, 2023 •

edited

Loading