HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. #5408

hfutatzhanghb · 2023-02-16T03:39:25Z

… to reduce impact on heartbeat

Hexiaoqiao · 2023-02-16T04:39:23Z

Update title and let's wait what will Yetus say.

hadoop-yetus · 2023-02-16T09:22:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	43m 11s		branch-3.3 passed
+1 💚	compile	1m 13s		branch-3.3 passed
+1 💚	checkstyle	0m 53s		branch-3.3 passed
+1 💚	mvnsite	1m 22s		branch-3.3 passed
+1 💚	javadoc	1m 33s		branch-3.3 passed
+1 💚	spotbugs	3m 21s		branch-3.3 passed
+1 💚	shadedclient	30m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 21s		the patch passed
+1 💚	compile	1m 9s		the patch passed
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	mvnsite	1m 18s		the patch passed
+1 💚	javadoc	1m 20s		the patch passed
+1 💚	spotbugs	3m 19s		the patch passed
+1 💚	shadedclient	30m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	221m 19s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		342m 3s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAuditLogs
	hadoop.hdfs.TestFileCreation
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.server.namenode.TestAuditLogger

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/1/artifact/out/Dockerfile
GITHUB PR	#5408
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2e9e1e2e533d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `db600f1`
Default Java	Private Build-1.8.0_352-8u352-ga-1~18.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/1/testReport/
Max. process+thread count	2360 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-02-16T09:44:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	45m 2s		branch-3.3 passed
+1 💚	compile	1m 25s		branch-3.3 passed
+1 💚	checkstyle	1m 1s		branch-3.3 passed
+1 💚	mvnsite	1m 36s		branch-3.3 passed
+1 💚	javadoc	1m 42s		branch-3.3 passed
+1 💚	spotbugs	3m 51s		branch-3.3 passed
+1 💚	shadedclient	32m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 37s		the patch passed
+1 💚	compile	1m 20s		the patch passed
+1 💚	javac	1m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	1m 24s		the patch passed
+1 💚	spotbugs	3m 40s		the patch passed
+1 💚	shadedclient	32m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	233m 6s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		362m 56s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAuditLogger
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.server.namenode.TestAuditLogs
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/2/artifact/out/Dockerfile
GITHUB PR	#5408
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 71030960ce81 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `db600f1`
Default Java	Private Build-1.8.0_352-8u352-ga-1~18.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/2/testReport/
Max. process+thread count	2025 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2023-02-16T09:50:55Z

@hfutatzhanghb Please check failed unit tests if relate with this changes. Thanks.

hfutatzhanghb · 2023-05-19T06:21:07Z

@hfutatzhanghb Please check failed unit tests if relate with this changes. Thanks.

@Hexiaoqiao Sir, the failed unit tests seems to have no relations with this PR.

hfutatzhanghb · 2023-06-07T02:27:57Z

@Hexiaoqiao sir, sorry for disturbing you here, could we push this issue forward? Thanks a lot , sir.

Hexiaoqiao · 2023-06-07T03:33:14Z

Retry to trigger Yetus manually, Let's wait what will it say.

hadoop-yetus · 2023-06-07T09:19:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	47m 35s		branch-3.3 passed
+1 💚	compile	1m 13s		branch-3.3 passed
+1 💚	checkstyle	1m 0s		branch-3.3 passed
+1 💚	mvnsite	1m 25s		branch-3.3 passed
+1 💚	javadoc	1m 40s		branch-3.3 passed
+1 💚	spotbugs	3m 45s		branch-3.3 passed
+1 💚	shadedclient	28m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	1m 8s		the patch passed
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	1m 17s		the patch passed
+1 💚	spotbugs	3m 20s		the patch passed
+1 💚	shadedclient	27m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	215m 24s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		346m 50s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestReconstructStripedFileWithValidator

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/3/artifact/out/Dockerfile
GITHUB PR	#5408
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 40d49d6d3828 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `db600f1`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/3/testReport/
Max. process+thread count	2232 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5408/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2023-06-08T11:58:37Z

@hfutatzhanghb Please check failed unit tests if relate to this changes.

hfutatzhanghb · 2023-06-10T13:39:04Z

@Hexiaoqiao Sir, the failed unit tests were all passed in my local.

Hexiaoqiao · 2023-06-12T04:23:37Z

Hi @hfutatzhanghb , Just notice that the target branch is branch-3.3 rather than trunk, any concerns?

hfutatzhanghb · 2023-06-12T04:55:29Z

Hi @hfutatzhanghb , Just notice that the target branch is branch-3.3 rather than trunk, any concerns?

@Hexiaoqiao Sir, this pr has merged into trunk, please see #5330.
and this page #5408 is just a backport to branch-3.3.

Hexiaoqiao

LGTM. +1.

Hexiaoqiao · 2023-06-12T05:42:57Z

Committed to branch-3.3. Thanks @hfutatzhanghb .

hfutatzhanghb · 2023-06-12T05:44:39Z

Committed to branch-3.3. Thanks @hfutatzhanghb .

Sir, thank for your reviewing again~

HDFS-16898. Remove write lock for processCommandFromActor of DataNode…

db600f1

… to reduce impact on heartbeat

hfutatzhanghb force-pushed the branch-3.3-processCMD branch from 106f85a to db600f1 Compare February 16, 2023 03:39

Hexiaoqiao changed the title ~~HDFS-16898.HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat~~ HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. Feb 16, 2023

Hexiaoqiao approved these changes Jun 12, 2023

View reviewed changes

Hexiaoqiao merged commit 0fd2b10 into apache:branch-3.3 Jun 12, 2023

slfan1989 mentioned this pull request Jan 15, 2024

HDFS-17129. mis-order of ibr and fbr on datanode #6244

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. #5408

HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. #5408

hfutatzhanghb commented Feb 16, 2023

Hexiaoqiao commented Feb 16, 2023

hadoop-yetus commented Feb 16, 2023

hadoop-yetus commented Feb 16, 2023

Hexiaoqiao commented Feb 16, 2023

hfutatzhanghb commented May 19, 2023

hfutatzhanghb commented Jun 7, 2023

Hexiaoqiao commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

Hexiaoqiao commented Jun 8, 2023

hfutatzhanghb commented Jun 10, 2023

Hexiaoqiao commented Jun 12, 2023

hfutatzhanghb commented Jun 12, 2023

Hexiaoqiao left a comment

Hexiaoqiao commented Jun 12, 2023

hfutatzhanghb commented Jun 12, 2023

HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. #5408

HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. #5408

Conversation

hfutatzhanghb commented Feb 16, 2023

Hexiaoqiao commented Feb 16, 2023

hadoop-yetus commented Feb 16, 2023

hadoop-yetus commented Feb 16, 2023

Hexiaoqiao commented Feb 16, 2023

hfutatzhanghb commented May 19, 2023

hfutatzhanghb commented Jun 7, 2023

Hexiaoqiao commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

Hexiaoqiao commented Jun 8, 2023

hfutatzhanghb commented Jun 10, 2023

Hexiaoqiao commented Jun 12, 2023

hfutatzhanghb commented Jun 12, 2023

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Hexiaoqiao commented Jun 12, 2023

hfutatzhanghb commented Jun 12, 2023