Add node id to segment and translog metadata #10229

gbbafna · 2023-09-26T03:59:27Z

Adding validation to identify multiple writers to same primary term and generation in remote store

Description

Adding node id to remote segment and translog metadata files . This is to identify if we have concurrent writers to same file , causing consistency issues.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-26T04:08:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26296/
CommitID: 0b971ce
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T04:37:37Z

Compatibility status:

Checks if related components are compatible with change 43fbb5d

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

github-actions · 2023-09-26T04:50:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26297/
CommitID: e1ff550
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T05:39:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26300/
CommitID: d38757b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T06:32:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26301/
CommitID: ff401e8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T08:54:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26306/
CommitID: 98a314c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T11:52:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26321/
CommitID: 98a314c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-27T04:22:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/26365/
CommitID: 98a314c
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-27T04:24:01Z

Codecov Report

Merging #10229 (43fbb5d) into main (d3bf230) will decrease coverage by 0.07%.
Report is 5 commits behind head on main.
The diff coverage is 89.39%.

@@             Coverage Diff              @@
##               main   #10229      +/-   ##
============================================
- Coverage     71.15%   71.09%   -0.07%     
+ Complexity    58247    58239       -8     
============================================
  Files          4831     4831              
  Lines        274684   274736      +52     
  Branches      40026    40034       +8     
============================================
- Hits         195443   195312     -131     
- Misses        62884    63085     +201     
+ Partials      16357    16339      -18

Files	Coverage Δ
...rch/extensions/rest/RestActionsRequestHandler.java	`100.00% <100.00%> (ø)`
...extensions/rest/RestInitializeExtensionAction.java	`65.00% <100.00%> (-0.44%)`	⬇️
...rch/extensions/rest/RestSendToExtensionAction.java	`42.14% <100.00%> (ø)`
...c/main/java/org/opensearch/index/IndexService.java	`75.65% <100.00%> (+0.05%)`	⬆️
.../org/opensearch/index/remote/RemoteStoreUtils.java	`96.66% <100.00%> (+2.22%)`	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.58% <100.00%> (-0.46%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`86.09% <100.00%> (+0.07%)`	⬆️
...earch/index/store/RemoteSegmentStoreDirectory.java	`89.93% <100.00%> (+0.25%)`	⬆️
...rg/opensearch/index/translog/RemoteFsTranslog.java	`72.59% <100.00%> (+1.09%)`	⬆️
.../org/opensearch/index/translog/TranslogConfig.java	`100.00% <100.00%> (ø)`
... and 6 more

... and 447 files with indirect coverage changes

Signed-off-by: Gaurav Bafna <[email protected]>

github-actions · 2023-10-03T10:44:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26685/
CommitID: 74bbe8d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gaurav Bafna <[email protected]>

github-actions · 2023-10-03T10:59:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26680/
CommitID: 825b9d0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-03T11:44:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.basic.SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions {p0={"search.concurrent_segment_search.enabled":"false"}}
      1 org.opensearch.search.basic.SearchWithRandomIOExceptionsIT.classMethod
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/26687/
CommitID: 43fbb5d
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/index/remote/RemoteStoreUtils.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

* Adding node id to segment and translog metadata Adding validation to identify multiple writers to same primary term and generation in remote store Signed-off-by: Gaurav Bafna <[email protected]> * Fix bug to find node id Signed-off-by: Gaurav Bafna <[email protected]> * Spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * Moving node id before uuid as it was interfering with _ character split Signed-off-by: Gaurav Bafna <[email protected]> * Removing uuid from segment metadata file Signed-off-by: Gaurav Bafna <[email protected]> * simplifying the detection logic Signed-off-by: Gaurav Bafna <[email protected]> * PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Addressing PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Adding translog gen to remote segment metadata fn as well Signed-off-by: Gaurav Bafna <[email protected]> * spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * reducing METADATA_FILES_TO_FETCH to 10 Signed-off-by: Gaurav Bafna <[email protected]> * adding missing import Signed-off-by: Gaurav Bafna <[email protected]> --------- Signed-off-by: Gaurav Bafna <[email protected]> (cherry picked from commit 7159e2e) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

ashking94 · 2023-10-04T10:44:52Z

server/src/main/java/org/opensearch/index/remote/RemoteStoreUtils.java

+                    throw new IllegalStateException(
+                        "Multiple metadata files from different nodes"
+                            + nodeIdByPrimaryTermAndGen.v1()
+                            + " and "
+                            + nodeIdByPrimaryTermAndGen.v2()
+                            + "having same primary term and generations detected"
+                    );


Thanks for adding more details

--------- (cherry picked from commit 7159e2e) Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Adding node id to segment and translog metadata Adding validation to identify multiple writers to same primary term and generation in remote store Signed-off-by: Gaurav Bafna <[email protected]> * Fix bug to find node id Signed-off-by: Gaurav Bafna <[email protected]> * Spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * Moving node id before uuid as it was interfering with _ character split Signed-off-by: Gaurav Bafna <[email protected]> * Removing uuid from segment metadata file Signed-off-by: Gaurav Bafna <[email protected]> * simplifying the detection logic Signed-off-by: Gaurav Bafna <[email protected]> * PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Addressing PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Adding translog gen to remote segment metadata fn as well Signed-off-by: Gaurav Bafna <[email protected]> * spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * reducing METADATA_FILES_TO_FETCH to 10 Signed-off-by: Gaurav Bafna <[email protected]> * adding missing import Signed-off-by: Gaurav Bafna <[email protected]> --------- Signed-off-by: Gaurav Bafna <[email protected]>

* Adding node id to segment and translog metadata Adding validation to identify multiple writers to same primary term and generation in remote store Signed-off-by: Gaurav Bafna <[email protected]> * Fix bug to find node id Signed-off-by: Gaurav Bafna <[email protected]> * Spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * Moving node id before uuid as it was interfering with _ character split Signed-off-by: Gaurav Bafna <[email protected]> * Removing uuid from segment metadata file Signed-off-by: Gaurav Bafna <[email protected]> * simplifying the detection logic Signed-off-by: Gaurav Bafna <[email protected]> * PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Addressing PR comments Signed-off-by: Gaurav Bafna <[email protected]> * Adding translog gen to remote segment metadata fn as well Signed-off-by: Gaurav Bafna <[email protected]> * spotless fix Signed-off-by: Gaurav Bafna <[email protected]> * reducing METADATA_FILES_TO_FETCH to 10 Signed-off-by: Gaurav Bafna <[email protected]> * adding missing import Signed-off-by: Gaurav Bafna <[email protected]> --------- Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

gbbafna added the skip-changelog label Sep 26, 2023

gbbafna force-pushed the dual-writer branch from 0b971ce to e1ff550 Compare September 26, 2023 04:15

gbbafna marked this pull request as ready for review September 27, 2023 04:40

gbbafna requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, peternied, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja and dreamer-89 as code owners September 27, 2023 04:40

gbbafna added 9 commits October 3, 2023 16:03

Spotless fix

e7cbe2b

Signed-off-by: Gaurav Bafna <[email protected]>

Moving node id before uuid as it was interfering with _ character split

ec8bac1

Signed-off-by: Gaurav Bafna <[email protected]>

Removing uuid from segment metadata file

65c54d2

Signed-off-by: Gaurav Bafna <[email protected]>

simplifying the detection logic

56f332e

Signed-off-by: Gaurav Bafna <[email protected]>

PR comments

3915b11

Signed-off-by: Gaurav Bafna <[email protected]>

Addressing PR comments

de097b3

Signed-off-by: Gaurav Bafna <[email protected]>

Adding translog gen to remote segment metadata fn as well

765194a

Signed-off-by: Gaurav Bafna <[email protected]>

spotless fix

db7ce52

Signed-off-by: Gaurav Bafna <[email protected]>

reducing METADATA_FILES_TO_FETCH to 10

74bbe8d

Signed-off-by: Gaurav Bafna <[email protected]>

gbbafna force-pushed the dual-writer branch from c362d5f to 74bbe8d Compare October 3, 2023 10:33

adding missing import

43fbb5d

Signed-off-by: Gaurav Bafna <[email protected]>

Bukhtawar reviewed Oct 4, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/remote/RemoteStoreUtils.java Show resolved Hide resolved

Bukhtawar reviewed Oct 4, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/IndexShard.java Show resolved Hide resolved

sachinpkale changed the title ~~Adding node id to segment and translog metadata~~ Add node id to segment and translog metadata Oct 4, 2023

sachinpkale approved these changes Oct 4, 2023

View reviewed changes

sachinpkale merged commit 7159e2e into opensearch-project:main Oct 4, 2023
21 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Oct 4, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 4, 2023

[Backport 2.x] Add node id to segment and translog metadata #10353

Merged

ashking94 reviewed Oct 4, 2023

View reviewed changes

rramachand21 added the v2.11.0 Issues and PRs related to version 2.11.0 label Oct 4, 2023

gbbafna mentioned this pull request Oct 6, 2023

[Remote Store] NodeId in segment and translog metadata file name may contain '-' #10450

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add node id to segment and translog metadata #10229

Add node id to segment and translog metadata #10229

gbbafna commented Sep 26, 2023 •

edited

Loading

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023 •

edited

Loading

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 27, 2023

codecov bot commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

ashking94 Oct 4, 2023

Add node id to segment and translog metadata #10229

Add node id to segment and translog metadata #10229

Conversation

gbbafna commented Sep 26, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 27, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 27, 2023 • edited Loading

Codecov Report

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 Oct 4, 2023

Choose a reason for hiding this comment

gbbafna commented Sep 26, 2023 •

edited

Loading

github-actions bot commented Sep 26, 2023 •

edited

Loading

codecov bot commented Sep 27, 2023 •

edited

Loading