-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update multipart download path to first write to temp files #10347
Conversation
Compatibility status:Checks if related components are compatible with change dfc8110 Incompatible componentsSkipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git] |
Gradle Check (Jenkins) Run Completed with:
|
My changes in #10349 will definitely conflict here, but it should be relatively easy to port the code. |
This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Marc Handalian <[email protected]>
.../src/main/java/org/opensearch/common/blobstore/stream/read/listener/ReadContextListener.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Marc Handalian <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #10347 +/- ##
============================================
- Coverage 71.09% 71.08% -0.01%
- Complexity 58209 58275 +66
============================================
Files 4830 4830
Lines 274840 274858 +18
Branches 40048 40048
============================================
+ Hits 195384 195385 +1
- Misses 63052 63116 +64
+ Partials 16404 16357 -47
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
The backport to
To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10347-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ca0dae61cb1845f2a2543d051a00be946635fc83
# Push it to GitHub
git push --set-upstream origin backport/backport-10347-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x Then, create a pull request where the |
…ch-project#10347) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit ca0dae6)
…10389) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit ca0dae6) Co-authored-by: Marc Handalian <[email protected]>
…ch-project#10347) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
…ch-project#10347) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
…ch-project#10347) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>
Description
This change updates ReadContextListener to first write parts to a temp location until all parts have been received. This ensures that partially written files do not lead to corruption. The temp location is prefixed with a random uuid ensuring there is no collision on retries.
Related Issues
Resolves #9784
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.