[Remote Store] Fix tests when we restore index without any refresh #9480

sachinpkale · 2023-08-22T13:23:24Z

Description

If we have an index where all the ingested data is in translog alone (the first refresh is not triggered yet), restoring such index from remote store fails due to issue mentioned in [BUG] Remote store restore using API fails when the index was created, docs indexed (optional) and node was stopped before the first refresh could happen #7923
This happens as the translog UUID of the downloaded translog files do not match with that of segment files (that are created newly as part of restore)
In this change, we make sure to create empty store with the same UUID that is present in the remote translog.

Related Issues

Resolves #7923

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sachin Kale <[email protected]>

server/src/main/java/org/opensearch/index/store/Store.java

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java

opensearch-trigger-bot · 2023-08-22T13:50:58Z

Compatibility status:

Checks if related components are compatible with change ebdffbb

Incompatible components

Incompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-22T14:03:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23203/
CommitID: f2b0cfb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreRestoreIT.java

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java

server/src/main/java/org/opensearch/index/store/Store.java

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale · 2023-08-23T04:59:27Z

The following tests are showing flaky behavior:

Tests with failures:
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testFullRestartDuringReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testDropRandomNodeDuringReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testBasicReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.classMethod

Checking if the flakiness is introduced with this PR.

sachinpkale · 2023-08-23T05:15:40Z

The following tests are showing flaky behavior:

Tests with failures:
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testFullRestartDuringReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testDropRandomNodeDuringReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.testBasicReplication
 - org.opensearch.indices.replication.SegmentReplicationSuiteIT.classMethod

Checking if the flakiness is introduced with this PR.

These are flaky tests and able to reproduce the failure on main as well. Issue: #9499

github-actions · 2023-08-23T05:32:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23265/
CommitID: ffe2d90

codecov · 2023-08-23T05:33:34Z

Codecov Report

Merging #9480 (ffe2d90) into main (9fb86a5) will decrease coverage by 0.17%.
Report is 5 commits behind head on main.
The diff coverage is 70.00%.

❗ Current head ffe2d90 differs from pull request most recent head dcff1e3. Consider uploading reports for the commit dcff1e3 to get more accurate results

@@             Coverage Diff              @@
##               main    #9480      +/-   ##
============================================
- Coverage     71.16%   71.00%   -0.17%     
+ Complexity    57478    57398      -80     
============================================
  Files          4776     4776              
  Lines        270811   270821      +10     
  Branches      39582    39582              
============================================
- Hits         192723   192288     -435     
- Misses        61902    62407     +505     
+ Partials      16186    16126      -60

Files Changed	Coverage Δ
...java/org/opensearch/index/shard/StoreRecovery.java	`57.06% <14.28%> (-0.19%)`	⬇️
...rc/main/java/org/opensearch/index/store/Store.java	`82.09% <100.00%> (+0.39%)`	⬆️
.../org/opensearch/index/translog/TranslogHeader.java	`81.48% <100.00%> (+3.27%)`	⬆️

... and 468 files with indirect coverage changes

sachinpkale · 2023-08-23T06:49:03Z

Added unit tests. StoreRecovery changes are covered in integ tests.

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreRestoreIT.java

server/src/test/java/org/opensearch/index/store/StoreTests.java

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java

Signed-off-by: Sachin Kale <[email protected]>

opensearch-trigger-bot · 2023-08-23T13:46:27Z

Compatibility status:

Checks if related components are compatible with change 5d3633c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

…9480) Signed-off-by: Sachin Kale <[email protected]> (cherry picked from commit 9272aa2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

github-actions · 2023-08-23T14:00:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23299/
CommitID: dcff1e3

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]>

…9480) (#9509) (cherry picked from commit 9272aa2) Signed-off-by: Sachin Kale <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Gagan Juneja <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Kiran Reddy <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

Fix tests when we restore index without any refresh

f2b0cfb

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale added the skip-changelog label Aug 22, 2023

sachinpkale commented Aug 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/Store.java Show resolved Hide resolved

sachinpkale commented Aug 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java Show resolved Hide resolved

BhumikaSaini-Amazon reviewed Aug 22, 2023

View reviewed changes

Add unit tests

ffe2d90

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale marked this pull request as ready for review August 23, 2023 04:44

sachinpkale requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and dbwiddis as code owners August 23, 2023 04:44

sachinpkale requested a review from sohami as a code owner August 23, 2023 04:44

sachinpkale requested a review from BhumikaSaini-Amazon August 23, 2023 04:44

BhumikaSaini-Amazon approved these changes Aug 23, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreRestoreIT.java Show resolved Hide resolved

server/src/test/java/org/opensearch/index/store/StoreTests.java Outdated Show resolved Hide resolved

gbbafna reviewed Aug 23, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/translog/TranslogHeader.java Show resolved Hide resolved

Address PR comments

dcff1e3

Signed-off-by: Sachin Kale <[email protected]>

gbbafna approved these changes Aug 23, 2023

View reviewed changes

gbbafna merged commit 9272aa2 into opensearch-project:main Aug 23, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Aug 23, 2023

opensearch-trigger-bot bot mentioned this pull request Aug 23, 2023

[Backport 2.x] [Remote Store] Fix tests when we restore index without any refresh #9509

Merged

neetikasinghal pushed a commit to neetikasinghal/OpenSearch that referenced this pull request Aug 23, 2023

[Remote Store] Fix tests when we restore index without any refresh (o…

53035aa

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]>

mch2 mentioned this pull request Aug 23, 2023

[BUG] Test org.opensearch.indices.replication.SegmentReplicationSuiteIT is flaky #9499

Closed

gbbafna pushed a commit that referenced this pull request Aug 24, 2023

[Remote Store] Fix tests when we restore index without any refresh (#…

02787cf

…9480) (#9509) (cherry picked from commit 9272aa2) Signed-off-by: Sachin Kale <[email protected]>

austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Aug 25, 2023

[Remote Store] Fix tests when we restore index without any refresh (o…

8765047

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]>

Gaganjuneja pushed a commit to Gaganjuneja/OpenSearch that referenced this pull request Aug 28, 2023

[Remote Store] Fix tests when we restore index without any refresh (o…

33f7926

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]>

kkmr pushed a commit to kkmr/OpenSearch that referenced this pull request Aug 28, 2023

[Remote Store] Fix tests when we restore index without any refresh (o…

0edd24f

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Kiran Reddy <[email protected]>

brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023

[Remote Store] Fix tests when we restore index without any refresh (o…

08e179c

…pensearch-project#9480) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Fix tests when we restore index without any refresh #9480

[Remote Store] Fix tests when we restore index without any refresh #9480

sachinpkale commented Aug 22, 2023 •

edited

Loading

opensearch-trigger-bot bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

sachinpkale commented Aug 23, 2023

sachinpkale commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

codecov bot commented Aug 23, 2023 •

edited

Loading

sachinpkale commented Aug 23, 2023

opensearch-trigger-bot bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

[Remote Store] Fix tests when we restore index without any refresh #9480

[Remote Store] Fix tests when we restore index without any refresh #9480

Conversation

sachinpkale commented Aug 22, 2023 • edited Loading

Description

Related Issues

Check List

opensearch-trigger-bot bot commented Aug 22, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 23, 2023

sachinpkale commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 23, 2023 • edited Loading

Codecov Report

sachinpkale commented Aug 23, 2023

opensearch-trigger-bot bot commented Aug 23, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 23, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 22, 2023 •

edited

Loading

codecov bot commented Aug 23, 2023 •

edited

Loading