HDFS-16146. All three replicas are lost due to not adding a new DataN… #3247
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
jojochuang
reviewed
Jul 30, 2021
| */ | ||
| if (!isAppend && lastAckedSeqno < 0 | ||
| && stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) { | ||
| //no data have been written | ||
| return; | ||
| } else if (stage == BlockConstructionStage.PIPELINE_CLOSE |
There was a problem hiding this comment.
the stage is in PIPELINE_CLOSE state when when packet is the last in the block and all transferred data is acknowledged.
with this change, the re-replication which was triggered by NameNode block manager periodically, is moved to client side.
it looks fine to me. but HDFS re replication logic is complex. We should validate it with a test.
There was a problem hiding this comment.
Thank you for your suggestion. I just added a UT to this patch.
Hexiaoqiao
reviewed
Aug 2, 2021
There was a problem hiding this comment.
Just leave some nits. Others LGTM. +1 from my side after fix the annotation.
@jojochuang Would you like to give another reviews? Will commit shortly if no other objections. Thanks.
| @@ -1492,6 +1492,8 @@ public void run() { | |||
| if (lastPacketInBlock) { | |||
| // Finalize the block and close the block file | |||
| finalizeBlock(startTime); | |||
| /* for test only, no-op in production system */ | |||
There was a problem hiding this comment.
Just suggest to use // annotation for single line code.
| @@ -68,6 +68,9 @@ public void delaySendingAckToUpstream(final String upstreamAddr) | |||
| throws IOException { | |||
| } | |||
|
|
|||
| public void delayAckLastPacket() throws IOException { | |||
There was a problem hiding this comment.
Please add Javadoc for this new injector method.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
approved these changes
Aug 3, 2021
|
looks good. @Hexiaoqiao please go ahead and merge it. thanks |
|
Committed to trunk. Thanks @zhangshuyan0 for your works. Thanks @jojochuang for your reviews. |
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
apache#3247) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]>
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
… a new DataN… (apache#3247) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit 10a2526) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeFaultInjector.java Change-Id: I47d8042771ed9dc634a866e0269989d253659a4e
LiuGuH
pushed a commit
to LiuGuH/hadoop
that referenced
this pull request
Mar 26, 2024
apache#3247) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]>
LiuGuH
pushed a commit
to LiuGuH/hadoop
that referenced
this pull request
Mar 29, 2024
apache#3247) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ode in time.
NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute