Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store #7394

Merged
merged 9 commits into from
May 15, 2023

Conversation

ankitkala
Copy link
Member

Description

This change adds a hook to publish the notification to replicas when a segments are uploaded to the remote store.
This change also gates this behind a new feature flag for Segment Replication with remote store..

Related Issues

#5578

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented May 3, 2023

Gradle Check (Jenkins) Run Completed with:

@ankitkala ankitkala force-pushed the remote_upload_hook branch 2 times, most recently from 1067906 to 511b605 Compare May 11, 2023 12:14
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented May 11, 2023

Codecov Report

Merging #7394 (e381c2c) into main (e44b3f1) will increase coverage by 0.09%.
The diff coverage is 71.42%.

@@             Coverage Diff              @@
##               main    #7394      +/-   ##
============================================
+ Coverage     70.47%   70.57%   +0.09%     
+ Complexity    59728    59726       -2     
============================================
  Files          4896     4896              
  Lines        286798   286807       +9     
  Branches      41331    41335       +4     
============================================
+ Hits         202133   202417     +284     
+ Misses        68078    67724     -354     
- Partials      16587    16666      +79     
Impacted Files Coverage Δ
.../main/java/org/opensearch/index/IndexSettings.java 84.19% <0.00%> (-0.44%) ⬇️
...in/java/org/opensearch/index/shard/IndexShard.java 70.64% <66.66%> (+0.58%) ⬆️
...nsearch/index/shard/CheckpointRefreshListener.java 91.66% <75.00%> (-8.34%) ⬇️
...search/index/shard/RemoteStoreRefreshListener.java 81.56% <100.00%> (-0.33%) ⬇️

... and 465 files with indirect coverage changes

…after segment upload to remote store

Signed-off-by: Ankit Kala <[email protected]>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <[email protected]>
@ankitkala ankitkala force-pushed the remote_upload_hook branch from cfc7f4c to e2b1a3b Compare May 15, 2023 09:40
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <[email protected]>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <[email protected]>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

Signed-off-by: Ankit Kala <[email protected]>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@mch2 mch2 merged commit 6f65afb into opensearch-project:main May 15, 2023
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request May 23, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <[email protected]>

* Added changelog

Signed-off-by: Ankit Kala <[email protected]>

* Fix the test

Signed-off-by: Ankit Kala <[email protected]>

* gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Addressed PR comments

Signed-off-by: Ankit Kala <[email protected]>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <[email protected]>

* Addressed comments 3

Signed-off-by: Ankit Kala <[email protected]>

* Fix Changelog entry

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request May 24, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <[email protected]>

* Added changelog

Signed-off-by: Ankit Kala <[email protected]>

* Fix the test

Signed-off-by: Ankit Kala <[email protected]>

* gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Addressed PR comments

Signed-off-by: Ankit Kala <[email protected]>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <[email protected]>

* Addressed comments 3

Signed-off-by: Ankit Kala <[email protected]>

* Fix Changelog entry

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
gbbafna pushed a commit that referenced this pull request May 24, 2023
…after segment upload to remote store (#7394) (#7718)

Signed-off-by: Ankit Kala <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <[email protected]>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <[email protected]>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <[email protected]>
dreamer-89 pushed a commit that referenced this pull request May 31, 2023
…cations after segment upload to remote store (#7394) (#7718)" (#7839)

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <[email protected]>
stephen-crawford pushed a commit to stephen-crawford/OpenSearch that referenced this pull request May 31, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <[email protected]>

* Added changelog

Signed-off-by: Ankit Kala <[email protected]>

* Fix the test

Signed-off-by: Ankit Kala <[email protected]>

* gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Addressed PR comments

Signed-off-by: Ankit Kala <[email protected]>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <[email protected]>

* Addressed comments 3

Signed-off-by: Ankit Kala <[email protected]>

* Fix Changelog entry

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <[email protected]>

* Added changelog

Signed-off-by: Ankit Kala <[email protected]>

* Fix the test

Signed-off-by: Ankit Kala <[email protected]>

* gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Addressed PR comments

Signed-off-by: Ankit Kala <[email protected]>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <[email protected]>

* Addressed comments 3

Signed-off-by: Ankit Kala <[email protected]>

* Fix Changelog entry

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants