Batch translog sync/upload per x ms for remote-backed indexes #5854

ashking94 · 2023-01-13T13:11:24Z

Description

Translog sync takes care of local fsync and translog upload onto remote store. Currently, there is implicit buffering that happens as the remote store upload is a time consuming operation. However, every upload adds extra cost of network interaction along with the actual file upload. If we can buffer for a pareto-optimal duration, then we can save on the additional network interaction costs and overall achieve lower latencies and higher indexing throughput in comparison to non-buffered approach. There is also a delay optimisation in place that make sure that if the upload took considerable time, then schedule the next run with a decreased interval maintaining the overall buffer interval in check.

Credits for some of the code - Ashwin Pankaj, Laxman Muttineni

Issues Resolved

This solves #5692

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-13T13:36:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9381/
CommitID: 6a2e359
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T17:31:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9386/
CommitID: 3f9cc1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-13T20:22:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/9396/
CommitID: b0d55c0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-16T09:48:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/9424/
CommitID: 07561b9
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-16T11:03:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9425/
CommitID: 776d2a2

codecov-commenter · 2023-01-16T11:05:48Z

Codecov Report

Merging #5854 (105cbc5) into main (715ff72) will decrease coverage by 0.04%.
The diff coverage is 75.60%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #5854      +/-   ##
============================================
- Coverage     70.75%   70.72%   -0.04%     
+ Complexity    58720    58704      -16     
============================================
  Files          4771     4772       +1     
  Lines        280818   280887      +69     
  Branches      40568    40572       +4     
============================================
- Hits         198704   198663      -41     
- Misses        65824    65860      +36     
- Partials      16290    16364      +74

Impacted Files	Coverage Δ
...pensearch/common/settings/IndexScopedSettings.java	`100.00% <ø> (ø)`
...earch/common/util/concurrent/AsyncIOProcessor.java	`92.98% <66.66%> (-1.02%)`	⬇️
...mmon/util/concurrent/BufferedAsyncIOProcessor.java	`68.96% <68.96%> (ø)`
...org/opensearch/cluster/metadata/IndexMetadata.java	`84.36% <76.92%> (-0.12%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.95% <78.94%> (-0.16%)`	⬇️
...n/java/org/opensearch/common/settings/Setting.java	`89.92% <100.00%> (+0.06%)`	⬆️
.../main/java/org/opensearch/index/IndexSettings.java	`86.20% <100.00%> (+0.11%)`	⬆️
...ain/java/org/opensearch/threadpool/ThreadPool.java	`83.48% <100.00%> (+0.69%)`	⬆️
...n/admin/cluster/node/tasks/get/GetTaskRequest.java	`30.30% <0.00%> (-63.64%)`	⬇️
...port/ResponseHandlerFailureTransportException.java	`0.00% <0.00%> (-60.00%)`	⬇️
... and 506 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-01-16T11:48:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9429/
CommitID: 6833b36
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-18T07:28:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDeleteOperations

URL: https://build.ci.opensearch.org/job/gradle-check/9521/
CommitID: 772addd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-19T08:27:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9581/
CommitID: ad03065
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T08:33:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9582/
CommitID: e1be6dd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T10:18:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9585/
CommitID: e70ce81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T13:03:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9591/
CommitID: e660f46
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-19T16:19:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9594/
CommitID: 32a73eb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-20T16:43:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testWriteRead
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testReadNonExistingPath
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testList
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testIndicesDeletedFromRepository
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testContainerCreationAndDeletion
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testDeleteOperations
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9646/
CommitID: 415a1cf
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Ashish Singh <[email protected]>

Co-authored-by: Ashwin Pankaj <[email protected]> Co-authored-by: Laxman Muttineni <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

Signed-off-by: Ashish Singh <[email protected]>

github-actions · 2023-01-25T08:57:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock

URL: https://build.ci.opensearch.org/job/gradle-check/9879/
CommitID: d455c21
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

andrross

Couple minor comments, otherwise looks good.

server/src/main/java/org/opensearch/index/IndexSettings.java

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java

Signed-off-by: Ashish Singh <[email protected]>

github-actions · 2023-01-28T18:50:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10113/
CommitID: 105cbc5

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java

opensearch-trigger-bot · 2023-01-29T04:36:35Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5854-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 af566e156fefdba192343ba8f7fce84f17d2a07a
# Push it to GitHub
git push --set-upstream origin backport/backport-5854-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5854-to-2.x.

opensearch-trigger-bot · 2023-01-30T08:10:55Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5854-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 af566e156fefdba192343ba8f7fce84f17d2a07a
# Push it to GitHub
git push --set-upstream origin backport/backport-5854-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5854-to-2.x.

…arch-project#5854) * Batch translog upload per x ms to allow high index throughput Signed-off-by: Ashish Singh <[email protected]> Co-authored-by: Ashwin Pankaj <[email protected]> Co-authored-by: Laxman Muttineni <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

…indexes (#5854) (#6066) * Batch translog sync/upload per x ms for remote-backed indexes (#5854) Signed-off-by: Ashish Singh <[email protected]> Co-authored-by: Ashwin Pankaj <[email protected]> Co-authored-by: Laxman Muttineni <[email protected]>

ashking94 self-assigned this Jan 13, 2023

ashking94 added Storage:Durability Issues and PRs related to the durability framework Performance This is for any performance related enhancements or bugs v2.6.0 'Issues and PRs related to version v2.6.0' distributed framework skip-changelog labels Jan 13, 2023

ashking94 force-pushed the 5692 branch from 6a2e359 to 3f9cc1d Compare January 13, 2023 17:03

ashking94 force-pushed the 5692 branch from 3f9cc1d to b0d55c0 Compare January 13, 2023 19:50

ashking94 changed the title ~~Batch translog upload per x ms to allow high index throughput~~ Batch translog upload per x ms for remote-backed indexes to allow high index throughput Jan 18, 2023

ashking94 changed the title ~~Batch translog upload per x ms for remote-backed indexes to allow high index throughput~~ Batch translog upload per x ms for remote-backed indexes Jan 18, 2023

ashking94 force-pushed the 5692 branch from 6833b36 to 772addd Compare January 18, 2023 06:52

ashking94 force-pushed the 5692 branch from ad03065 to e1be6dd Compare January 19, 2023 08:18

ashking94 force-pushed the 5692 branch from e1be6dd to e70ce81 Compare January 19, 2023 09:47

ashking94 force-pushed the 5692 branch from e70ce81 to e660f46 Compare January 19, 2023 12:34

ashking94 force-pushed the 5692 branch from e660f46 to 32a73eb Compare January 19, 2023 15:50

ashking94 force-pushed the 5692 branch from 32a73eb to 415a1cf Compare January 20, 2023 16:10

ashking94 and others added 8 commits January 25, 2023 13:54

Make buffer_interval setting Final

8a14c89

Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

e352565

Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

1f1ed7d

Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

70558bd

Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

3a5f960

Co-authored-by: Ashwin Pankaj <[email protected]> Co-authored-by: Laxman Muttineni <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

9dd5d88

Signed-off-by: Ashish Singh <[email protected]>

Incorporate PR review feedback

2ddda27

Signed-off-by: Ashish Singh <[email protected]>

Handle scheduling failures

d455c21

Signed-off-by: Ashish Singh <[email protected]>

ashking94 force-pushed the 5692 branch from 0c9105b to d455c21 Compare January 25, 2023 08:28

sachinpkale approved these changes Jan 25, 2023

View reviewed changes

andrross reviewed Jan 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/IndexSettings.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java Outdated Show resolved Hide resolved

Incorporate PR review feedback

105cbc5

Signed-off-by: Ashish Singh <[email protected]>

gbbafna approved these changes Jan 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/common/util/concurrent/BufferedAsyncIOProcessor.java Show resolved Hide resolved

gbbafna merged commit af566e1 into opensearch-project:main Jan 29, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Jan 29, 2023

ashking94 mentioned this pull request Jan 30, 2023

Refactor buffer interval code for remote-backed indexes #6063

Merged

6 tasks

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jan 30, 2023

ashking94 mentioned this pull request Jan 30, 2023

[Backport 2.x] Batch translog sync/upload per x ms for remote-backed indexes (#5854) #6066

Merged

6 tasks

sachinpkale mentioned this pull request Feb 4, 2023

[BUG] RemoteStoreIT.testRemoteTranslogRestore flaky test failure #6188

Closed

sachinpkale mentioned this pull request Feb 21, 2023

[Remote Translog] Add missing changelog entries #6408

Closed

6 tasks

ashking94 mentioned this pull request Mar 8, 2024

Optimised prefix pattern per shard for remote store data and metadata files for higher throughput #12567

Closed

Bukhtawar mentioned this pull request Jun 27, 2024

Add Ashish Singh as maintainer #14567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch translog sync/upload per x ms for remote-backed indexes #5854

Batch translog sync/upload per x ms for remote-backed indexes #5854

ashking94 commented Jan 13, 2023 •

edited

Loading

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

codecov-commenter commented Jan 16, 2023 •

edited

Loading

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

github-actions bot commented Jan 20, 2023

github-actions bot commented Jan 25, 2023

andrross left a comment

github-actions bot commented Jan 28, 2023

opensearch-trigger-bot bot commented Jan 29, 2023

opensearch-trigger-bot bot commented Jan 30, 2023

Batch translog sync/upload per x ms for remote-backed indexes #5854

Batch translog sync/upload per x ms for remote-backed indexes #5854

Conversation

ashking94 commented Jan 13, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jan 16, 2023 • edited Loading

Codecov Report

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

andrross left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 28, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jan 29, 2023

opensearch-trigger-bot bot commented Jan 30, 2023

ashking94 commented Jan 13, 2023 •

edited

Loading

codecov-commenter commented Jan 16, 2023 •

edited

Loading