HDFS-16423. Balancer should not get blocks on stale storages #3883
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liubingxing
force-pushed
the
HDFS-16423
branch
2 times, most recently
from
2159b9d to
30c8dce
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
liubingxing
force-pushed
the
HDFS-16423
branch
2 times, most recently
from
2b4793d to
11ef2f0
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
liubingxing
force-pushed
the
HDFS-16423
branch
from
11ef2f0 to
3d35dae
Compare
|
💔 -1 overall
This message was automatically generated. |
liubingxing
force-pushed
the
HDFS-16423
branch
from
3d35dae to
fd3415a
Compare
|
💔 -1 overall
This message was automatically generated. |
|
@tasanuma @Hexiaoqiao Please take a look. |
|
@liubingxing Thanks for the PR. It is a kind of bug, and I don't think we need to make it configurable. |
liubingxing
force-pushed
the
HDFS-16423
branch
from
fd3415a to
51eeeb4
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
@tasanuma Thanks for your comment, I update the PR, Please take a look. |
tomscut
approved these changes
Jan 18, 2022
tomscut
reviewed
Jan 18, 2022
| iter = node.getBlockIterator(); // start from the beginning | ||
| for(int i=0; i<startBlock&&totalSize<size; i++) { | ||
| iter = node.getBlockIterator(0, storageInfos); // start from the beginning | ||
| for(int i=0; i<startBlock&&totalSize<size&&iter.hasNext(); i++) { |
There was a problem hiding this comment.
Please add blank spaces to correct the code format. Thanks.
There was a problem hiding this comment.
Thanks for the advice, I update the code, please review again.
liubingxing
force-pushed
the
HDFS-16423
branch
from
51eeeb4 to
8d4b69d
Compare
tomscut
approved these changes
Jan 19, 2022
|
+1, pending Jenkins. |
|
🎊 +1 overall
This message was automatically generated. |
tasanuma
approved these changes
Jan 19, 2022
|
Merged it. Thanks for your contribution, @liubingxing, and thanks for your review, @tomscut. |
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
Jan 25, 2022
…3883) Reviewed-by: litao <[email protected]> Signed-off-by: Takanobu Asanuma <[email protected]> (cherry picked from commit db2c320) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java
jojochuang
added a commit
that referenced
this pull request
Jan 26, 2022
…3924) Reviewed-by: litao <[email protected]> Signed-off-by: Takanobu Asanuma <[email protected]> (cherry picked from commit db2c320) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java Co-authored-by: qinyuren <[email protected]>
bogthe
pushed a commit
to bogthe/hadoop
that referenced
this pull request
Feb 2, 2022
…3883) (apache#3924) Reviewed-by: litao <[email protected]> Signed-off-by: Takanobu Asanuma <[email protected]> (cherry picked from commit db2c320) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java Co-authored-by: qinyuren <[email protected]>
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…3883) Reviewed-by: litao <[email protected]> Signed-off-by: Takanobu Asanuma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: HDFS-16423
We have met a problems as described in HDFS-16420
We found that balancer copied a block multi times without deleting the source block if this block was placed in a stale storage. And resulting a block with many copies, but these redundant copies are not deleted until the storage become not stale.
