-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Segment Replication] Introduce primary weight factor for primary shards distribution #6017
[Segment Replication] Introduce primary weight factor for primary shards distribution #6017
Conversation
Gradle Check (Jenkins) Run Completed with:
|
...c/main/java/org/opensearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java
Outdated
Show resolved
Hide resolved
00e8900
to
fee896f
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #6017 +/- ##
============================================
- Coverage 70.87% 70.72% -0.15%
+ Complexity 58830 58752 -78
============================================
Files 4776 4778 +2
Lines 280993 281074 +81
Branches 40598 40617 +19
============================================
- Hits 199141 198794 -347
- Misses 65488 65852 +364
- Partials 16364 16428 +64
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
…aration Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
92e9713
to
11b0c80
Compare
Gradle Check (Jenkins) Run Completed with:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @dreamer-89 for this change!
…rds distribution (#6017) * Add integration test to show shard allocation Signed-off-by: Suraj Singh <[email protected]> * Update WeightFunction to consider primary shards for uniform primary distribution Signed-off-by: Suraj Singh <[email protected]> * Include primary shard weight for all shard types Signed-off-by: Suraj Singh <[email protected]> * Update integration test to show docrep & segrep indices Signed-off-by: Suraj Singh <[email protected]> * Add settings updater and update TriConsumer functional interface declaration Signed-off-by: Suraj Singh <[email protected]> * Add balance configuration test Signed-off-by: Suraj Singh <[email protected]> * Fix failing unit tests and merge conflicts Signed-off-by: Suraj Singh <[email protected]> * Rename primary balance factor Signed-off-by: Suraj Singh <[email protected]> * Update integration tests to avoid green state timeouts Signed-off-by: Suraj Singh <[email protected]> * PR feedback Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]> (cherry picked from commit aa21b5f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…rds distribution (#6017) (#6161) * Add integration test to show shard allocation * Update WeightFunction to consider primary shards for uniform primary distribution * Include primary shard weight for all shard types * Update integration test to show docrep & segrep indices * Add settings updater and update TriConsumer functional interface declaration * Add balance configuration test * Fix failing unit tests and merge conflicts * Rename primary balance factor * Update integration tests to avoid green state timeouts * PR feedback --------- (cherry picked from commit aa21b5f) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…mary shards distribution (opensearch-project#6017) (opensearch-project#6161)" This reverts commit 2e3e1ed.
…mary shards distribution (opensearch-project#6017)" This reverts commit aa21b5f.
* Revert "[Segment Replication] Add changelog entry and fix allocation test (#6250) (#6270)" This reverts commit 726c068. * Revert "[Segment Replication] Introduce primary weight factor for primary shards distribution (#6017) (#6161)" This reverts commit 2e3e1ed. (cherry picked from commit e316959) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…6444) * Revert "[Segment Replication] Add changelog entry and fix allocation test (#6250) (#6270)" This reverts commit 726c068. * Revert "[Segment Replication] Introduce primary weight factor for primary shards distribution (#6017) (#6161)" This reverts commit 2e3e1ed. (cherry picked from commit e316959) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…rds distribution (opensearch-project#6017) * Add integration test to show shard allocation Signed-off-by: Suraj Singh <[email protected]> * Update WeightFunction to consider primary shards for uniform primary distribution Signed-off-by: Suraj Singh <[email protected]> * Include primary shard weight for all shard types Signed-off-by: Suraj Singh <[email protected]> * Update integration test to show docrep & segrep indices Signed-off-by: Suraj Singh <[email protected]> * Add settings updater and update TriConsumer functional interface declaration Signed-off-by: Suraj Singh <[email protected]> * Add balance configuration test Signed-off-by: Suraj Singh <[email protected]> * Fix failing unit tests and merge conflicts Signed-off-by: Suraj Singh <[email protected]> * Rename primary balance factor Signed-off-by: Suraj Singh <[email protected]> * Update integration tests to avoid green state timeouts Signed-off-by: Suraj Singh <[email protected]> * PR feedback Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>
Description
This change introduces the mechanism to balance primary shards across cluster
PRIMARY_BALANCE_FACTOR_SETTING
setting which defines the constant for primary shards; used to calculate the weight on node. This factor defines the tendency to balance the primary shards; higher the value more chances of even distribution. This setting's default is initially set to0.0f
to avoid rebalance disruption in existing clusters. The other settings used in weight calculation areINDEX_BALANCE_FACTOR_SETTING
govern shards/node specific to index and SHARD_BALANCE_FACTOR_SETTING shards/node.TriConsumer
to conform to existing Java Consumer interface definition and related unit tests changes. It used as consumer to reflect settings changes to WeightFunction.Default setting consideration
PRIMARY_BALANCE_FACTOR_SETTING
is initially set to0
to prevent unnecessary disruption in doc rep indices. One improvement is to use some sane default when there are segrep enabled indices to automatically balance cluster based on primary shard. This is difficult asIssues Resolved
#5240
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.