HDFS-16348. Mark slownode as badnode to recover pipeline #3704

symious · 2021-11-22T14:50:06Z

Description of PR

In HDFS-16320, the DataNode can retrieve the SLOW status from each NameNode.

This ticket is to send this information back to Clients who are writing blocks. If a Clients noticed the pipeline is build on a slownode, he/she can choose to mark the slownode as a badnode to exclude the node or rebuild a pipeline.

In order to avoid the false positives, we added a config of "threshold", only clients continuously receives slownode reply from the same node will the node be marked as SLOW.

Jira ticket: https://issues.apache.org/jira/browse/HDFS-16348

How was this patch tested?

unit test

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

symious · 2021-11-22T14:50:45Z

@ferhui @tasanuma @Hexiaoqiao Could you help to review this PR?

hadoop-yetus · 2021-11-22T20:47:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 32s		trunk passed
+1 💚	compile	5m 19s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 1s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 17s		trunk passed
+1 💚	mvnsite	2m 28s		trunk passed
+1 💚	javadoc	1m 45s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 38s		trunk passed
+1 💚	shadedclient	22m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 6s		the patch passed
+1 💚	compile	5m 9s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 9s		the patch passed
+1 💚	compile	4m 52s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	4m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 8s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 3 new + 378 unchanged - 1 fixed = 381 total (was 379)
+1 💚	mvnsite	2m 10s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 26s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 0s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 43s		the patch passed
+1 💚	shadedclient	21m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 21s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	225m 50s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		356m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/1/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 8b50538bfabd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d7d2dc4`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/1/testReport/
Max. process+thread count	2818 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-25T15:24:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 24s		trunk passed
+1 💚	compile	6m 8s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 27s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 16s		trunk passed
+1 💚	mvnsite	2m 23s		trunk passed
+1 💚	javadoc	1m 39s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 3s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 56s		trunk passed
+1 💚	shadedclient	23m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 12s		the patch passed
+1 💚	compile	5m 44s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 44s		the patch passed
+1 💚	compile	5m 10s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 9s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 3 new + 381 unchanged - 1 fixed = 384 total (was 382)
+1 💚	mvnsite	2m 12s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 27s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 56s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 4s		the patch passed
+1 💚	shadedclient	23m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 25s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	232m 0s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		367m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/2/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux daf487591f77 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e323a1`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/2/testReport/
Max. process+thread count	3869 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma · 2021-12-08T06:24:43Z

@symious Sorry for reviewing it very late. I looked through it, and it mostly looks good to me.

I have one question. This feature is similar to ECN. The configuration of ECN is dfs.pipeline.ecn, which is for the DataNode configuration. dfs.client.mark.slownode.as.badnode seems the client configuration. If it works the same way as ECN, it would be better to make it on the DataNode configuration. What do you think?

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java

symious · 2021-12-08T14:12:17Z

@tasanuma Thanks for the review.

I have one question. This feature is similar to ECN. The configuration of ECN is dfs.pipeline.ecn, which is for the DataNode configuration. dfs.client.mark.slownode.as.badnode seems the client configuration

I tried to digest the design of the code again. IMHO, the config of "dfs.pipeline.ecn" is like "dfs.pipeline.ecn.enable", the value of "true" means datanode should add the ECN information in PipelineAck, so this configuration is for DataNode to add or not add ECN. On Client side, there is no config for clients to choose if she should use ECN information, since it will use the information by default.

In this ticket, I think a similar config of "dfs.pipeline.ecn" should be "dfs.pipeline.reply_slownode_in_pipeline_ack.enable", based on which DataNodes will choose to add or skip the slownode information in PipelineAck.
On Client side, "dfs.client.mark.slownode.as.badnode" is more like "dfs.client.handle.ECN_from_pipelineACK".

I think it's more depend on the subject of the operation, that is the subject of "dfs.pipeline.ecn" is DataNode, while the subject of "dfs.client.mark.slownode.as.badnode" is Client.

tasanuma · 2021-12-10T17:29:37Z

@symious Thanks for your explanation.
So my question is which do you think better to implement the configuration, on the client side (dfs.client.mark.slownode.as.badnode) or on the datanode side (dfs.pipeline.reply_slownode_in_pipeline_ack.enable)? If this feature is similar to ECN, this ticket would be better to implement dfs.pipeline.reply_slownode_in_pipeline_ack.enable. Or do you think dfs.client.mark.slownode.as.badnode is better?

symious · 2021-12-12T12:20:59Z

@tasanuma Sorry for the late reply.

If this feature is similar to ECN, this ticket would be better to implement dfs.pipeline.reply_slownode_in_pipeline_ack.enable. Or do you think dfs.client.mark.slownode.as.badnode is better?

Since it's similar to ECN, I think it's reasonable to set on the server side.

Another problem I was thinking is the configuration of "dfs.client.mark.slownode.as.badnode.threshold", this configuration seems more related to the choice of Clients, it might be a little rigid to set a default value on server side. Should I leave this configuration on client side?

tasanuma · 2021-12-12T13:02:05Z

@symious Thanks for your reply.

Since it's similar to ECN, I think it's reasonable to set on the server side.

I agreed. Thanks.

Another problem I was thinking is the configuration of "dfs.client.mark.slownode.as.badnode.threshold", this configuration seems more related to the choice of Clients, it might be a little rigid to set a default value on server side. Should I leave this configuration on client side?

I also agreed that dfs.client.mark.slownode.as.badnode.threshold is more related to clients. Let's leave it on the client side.

hadoop-yetus · 2021-12-12T18:57:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 47s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 42s		trunk passed
+1 💚	compile	5m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 15s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	2m 50s		trunk passed
+1 💚	javadoc	2m 4s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 30s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 9s		trunk passed
+1 💚	shadedclient	30m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 37s		the patch passed
+1 💚	compile	6m 57s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	6m 57s		the patch passed
+1 💚	compile	6m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	6m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 26s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 327 unchanged - 1 fixed = 328 total (was 328)
+1 💚	mvnsite	2m 29s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 42s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 23s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 22s		the patch passed
+1 💚	shadedclient	29m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 32s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	244m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		399m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecoveryStriped
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/3/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 4428dba61480 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `54045ae`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/3/testReport/
Max. process+thread count	3370 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-13T07:52:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 57s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 33s		trunk passed
+1 💚	compile	5m 23s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 5s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	2m 26s		trunk passed
+1 💚	javadoc	1m 45s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 39s		trunk passed
+1 💚	shadedclient	22m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 3s		the patch passed
+1 💚	compile	5m 14s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 14s		the patch passed
+1 💚	compile	5m 2s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 6s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 327 unchanged - 1 fixed = 328 total (was 328)
+1 💚	mvnsite	2m 7s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 27s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 2s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 50s		the patch passed
+1 💚	shadedclient	22m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 24s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	224m 41s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		354m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/4/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux f06149a5f0dc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `df061ad`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/4/testReport/
Max. process+thread count	3223 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-15T15:55:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 59s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 51s		trunk passed
+1 💚	compile	5m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 1s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 18s		trunk passed
+1 💚	mvnsite	2m 24s		trunk passed
+1 💚	javadoc	1m 46s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 37s		trunk passed
+1 💚	shadedclient	22m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 11s		the patch passed
+1 💚	compile	5m 37s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 37s		the patch passed
+1 💚	compile	5m 8s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 5s		hadoop-hdfs-project: The patch generated 0 new + 528 unchanged - 1 fixed = 528 total (was 529)
+1 💚	mvnsite	2m 12s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 41s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 35s		the patch passed
+1 💚	shadedclient	30m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 35s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	237m 7s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		384m 49s

Reason	Tests
Failed junit tests	hadoop.tools.TestHdfsConfigFields
	hadoop.hdfs.TestErasureCodeBenchmarkThroughput

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/5/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux b88dbf38ba3a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `020093e`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/5/testReport/
Max. process+thread count	3410 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-16T21:34:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 8s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 31s		trunk passed
+1 💚	compile	6m 11s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 52s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	2m 30s		trunk passed
+1 💚	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 7s		trunk passed
+1 💚	shadedclient	24m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 9s		the patch passed
+1 💚	compile	5m 33s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 33s		the patch passed
+1 💚	compile	5m 2s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 8s		hadoop-hdfs-project: The patch generated 0 new + 528 unchanged - 1 fixed = 528 total (was 529)
+1 💚	mvnsite	2m 15s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 29s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 55s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 57s		the patch passed
+1 💚	shadedclient	22m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 23s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	228m 36s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		367m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/6/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 858276b9068a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e8b280`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/6/testReport/
Max. process+thread count	3285 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

symious · 2021-12-20T02:41:35Z

@tasanuma Updated the patch, could you help to check?

tasanuma

@symious Thanks for updating PR. It mostly looks good. I left some minor comments.

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java

...doop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java

...-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java

symious · 2021-12-21T05:33:08Z

@tasanuma Thanks for the detailed review. Updated as suggested, please have a check.

tasanuma · 2021-12-21T09:16:17Z

Thanks for updating PR, @symious. +1 from me, pending Jenkins.

hadoop-yetus · 2021-12-21T13:16:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 42s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 52s		trunk passed
+1 💚	compile	5m 59s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 39s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	2m 22s		trunk passed
+1 💚	javadoc	1m 40s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 2s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 53s		trunk passed
+1 💚	shadedclient	25m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s		the patch passed
+1 💚	compile	5m 53s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 53s		the patch passed
+1 💚	compile	5m 27s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 9s		hadoop-hdfs-project: The patch generated 0 new + 528 unchanged - 1 fixed = 528 total (was 529)
+1 💚	mvnsite	2m 10s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 26s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 53s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 9s		the patch passed
+1 💚	shadedclient	25m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 18s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	320m 50s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		462m 26s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/7/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux dd4f5befacaf 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6a6875d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/7/testReport/
Max. process+thread count	2257 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-21T21:15:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 32s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 29s		trunk passed
+1 💚	compile	6m 6s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 32s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 16s		trunk passed
+1 💚	mvnsite	2m 20s		trunk passed
+1 💚	javadoc	1m 40s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 6s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 55s		trunk passed
+1 💚	shadedclient	25m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 8s		the patch passed
+1 💚	compile	5m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 54s		the patch passed
+1 💚	compile	5m 26s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 12s		hadoop-hdfs-project: The patch generated 0 new + 528 unchanged - 1 fixed = 528 total (was 529)
+1 💚	mvnsite	2m 11s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 27s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 57s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 6s		the patch passed
+1 💚	shadedclient	25m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 17s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	313m 54s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		455m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/8/artifact/out/Dockerfile
GITHUB PR	#3704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 180c2a1af2dc 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `97485b9`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/8/testReport/
Max. process+thread count	2358 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3704/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma

LGTM. I will merge it in a week if there is no objection.

tasanuma · 2021-12-29T15:28:07Z

Merged it. Thanks for your great work, @symious!

HDFS-16348. Mark slownode as badnode to recover pipeline

d7d2dc4

HDFS-16348. Add pipeline recovery test after mark slownode as badnode

3e323a1

tasanuma reviewed Dec 8, 2021

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java Outdated Show resolved Hide resolved

HDFS-16348. Reuse map in DataStream

54045ae

HDFS-16348. Try to fix methodTooLong issue

df061ad

HDFS-16348. Change to dfs.pipeline.slownode

020093e

HDFS-16348. Remote unused config from hdfs-default.xml

3e8b280

tasanuma reviewed Dec 21, 2021

View reviewed changes

HDDS-16348. Fix comment suggestions

6a6875d

trigger new CI check

97485b9

tasanuma approved these changes Dec 22, 2021

View reviewed changes

tasanuma merged commit b225287 into apache:trunk Dec 29, 2021

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16348. Mark slownode as badnode to recover pipeline (apache#3704)

43e6c65

xleoken mentioned this pull request Mar 27, 2024

HDFS-17397. Choose another DN as soon as possible, when encountering network issues #6591

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16348. Mark slownode as badnode to recover pipeline #3704

HDFS-16348. Mark slownode as badnode to recover pipeline #3704

symious commented Nov 22, 2021

symious commented Nov 22, 2021

hadoop-yetus commented Nov 22, 2021

hadoop-yetus commented Nov 25, 2021

tasanuma commented Dec 8, 2021

symious commented Dec 8, 2021

tasanuma commented Dec 10, 2021

symious commented Dec 12, 2021

tasanuma commented Dec 12, 2021

hadoop-yetus commented Dec 12, 2021

hadoop-yetus commented Dec 13, 2021

hadoop-yetus commented Dec 15, 2021

hadoop-yetus commented Dec 16, 2021

symious commented Dec 20, 2021

tasanuma left a comment

symious commented Dec 21, 2021

tasanuma commented Dec 21, 2021

hadoop-yetus commented Dec 21, 2021

hadoop-yetus commented Dec 21, 2021

tasanuma left a comment

tasanuma commented Dec 29, 2021

HDFS-16348. Mark slownode as badnode to recover pipeline #3704

HDFS-16348. Mark slownode as badnode to recover pipeline #3704

Conversation

symious commented Nov 22, 2021

Description of PR

How was this patch tested?

For code changes:

symious commented Nov 22, 2021

hadoop-yetus commented Nov 22, 2021

hadoop-yetus commented Nov 25, 2021

tasanuma commented Dec 8, 2021

symious commented Dec 8, 2021

tasanuma commented Dec 10, 2021

symious commented Dec 12, 2021

tasanuma commented Dec 12, 2021

hadoop-yetus commented Dec 12, 2021

hadoop-yetus commented Dec 13, 2021

hadoop-yetus commented Dec 15, 2021

hadoop-yetus commented Dec 16, 2021

symious commented Dec 20, 2021

tasanuma left a comment

Choose a reason for hiding this comment

symious commented Dec 21, 2021

tasanuma commented Dec 21, 2021

hadoop-yetus commented Dec 21, 2021

hadoop-yetus commented Dec 21, 2021

tasanuma left a comment

Choose a reason for hiding this comment

tasanuma commented Dec 29, 2021