Add file cache restore logic #6538
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kotwanikunal
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
mch2,
nknize,
owaiskazi19,
adnapibar,
Rishikesh1159,
ryanbogan,
saratvemulapalli,
shwetathareja,
dreamer-89,
tlfeng,
VachaShah and
xuezhou25
as code owners
Gradle Check (Jenkins) Run Completed with:
|
andrross
reviewed
Mar 3, 2023
server/src/main/java/org/opensearch/index/store/remote/filecache/FileCache.java
Outdated
Show resolved
Hide resolved
| if (subPathFile.isFile()) { | ||
| fileCache.put( | ||
| subPathFile.toPath().toRealPath(), | ||
| new FileCachedIndexInput.ClosedIndexInput(subPathFile.length()) |
There was a problem hiding this comment.
This is so gross, but I understand why you're doing it. I think we should be caching a pointer to a chunk on disk instead of an InputIndex (we can create an IndexInput for the file chunk on demand when it is needed) but that is a bigger cache refactoring discussion.
There was a problem hiding this comment.
Should we create an issue to track this?
kotwanikunal
force-pushed
the
cache-restore
branch
from
cc4368c to
8b8d5c4
Compare
Gradle Check (Jenkins) Run Completed with:
|
andrross
reviewed
Mar 8, 2023
server/src/main/java/org/opensearch/index/store/remote/filecache/FileCache.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/opensearch/index/store/remote/filecache/FileCacheTests.java
Outdated
Show resolved
Hide resolved
kotwanikunal
force-pushed
the
cache-restore
branch
from
8b8d5c4 to
97592fc
Compare
kotwanikunal
force-pushed
the
cache-restore
branch
from
97592fc to
cd5c4f6
Compare
Gradle Check (Jenkins) Run Completed with:
|
andrross
reviewed
Mar 8, 2023
server/src/main/java/org/opensearch/index/store/remote/filecache/FileCache.java
Outdated
Show resolved
Hide resolved
Gradle Check (Jenkins) Run Completed with:
|
kotwanikunal
force-pushed
the
cache-restore
branch
from
cd5c4f6 to
0ac19cc
Compare
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #6538 +/- ##
============================================
+ Coverage 70.77% 70.80% +0.03%
+ Complexity 59125 59106 -19
============================================
Files 4804 4804
Lines 283098 283128 +30
Branches 40813 40813
============================================
+ Hits 200372 200481 +109
+ Misses 66269 66206 -63
+ Partials 16457 16441 -16
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
andrross
reviewed
Mar 9, 2023
| } | ||
|
|
||
| NodesStatsResponse postRestoreStats = client().admin().cluster().nodesStats(new NodesStatsRequest().all()).actionGet(); | ||
| java.util.Map<String, NodeStats> preRestoreStatsMap = preRestoreStats.getNodesMap(); |
There was a problem hiding this comment.
Nitpick, but you can you change the import on line 27 use java.util.Map so that you don't have to do the fully qualified thing here?
kotwanikunal
force-pushed
the
cache-restore
branch
from
0ac19cc to
c25a4d9
Compare
Signed-off-by: Kunal Kotwani <[email protected]>
kotwanikunal
force-pushed
the
cache-restore
branch
from
c25a4d9 to
c034a88
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
andrross
approved these changes
Mar 9, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Mar 9, 2023
Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 16535b9) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kotwanikunal
pushed a commit
that referenced
this pull request
Mar 9, 2023
(cherry picked from commit 16535b9) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mingshl
pushed a commit
to mingshl/OpenSearch-Mingshl
that referenced
this pull request
Mar 24, 2023
Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Mingshi Liu <[email protected]>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Issues Resolved
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.