HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic #6245

haiyang1987 · 2023-11-02T11:12:34Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-17247

The AvailableSpaceBlockPlacementPolicy has made some optimizations such as HDFS-14578 can have local node optimization and HDFS-17210 can avoid choosing nodes with higher storage, the AvailableSpaceRackFaultTolerantBlockPlacementPolicy need keep same logics of optimization same as ASBPP.
Currently considering implementing a public class, such as AvailableSpaceBlockPlacementPolicyUtils, and abstract some public methods such as chooseLocalStorage, swapStorageTypes, select, compareDataNode, the AvailableSpaceBlockPlacementPolicy and AvailableSpaceRackFaultTolerantBlockPlacementPolicy can call related methods in AvailableSpaceBlockPlacementPolicyUtils. If there are related optimizations later, they can be modified based on AvailableSpaceBlockPlacementPolicyUtils.
Improve some doc descriptions and configuration parameters

…cy logic

hadoop-yetus · 2023-11-02T16:16:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	39m 34s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 46s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 11s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 56s		trunk passed
+1 💚	shadedclient	22m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 43s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 40s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 35s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 22 new + 228 unchanged - 6 fixed = 250 total (was 234)
+1 💚	mvnsite	0m 46s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 7s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 54s		the patch passed
+1 💚	shadedclient	21m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	195m 8s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		303m 7s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSUtil

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/1/artifact/out/Dockerfile
GITHUB PR	#6245
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux e98ad354d786 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a988804`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/1/testReport/
Max. process+thread count	3776 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2023-11-04T03:29:55Z

The implementation logic in both the AvailableSpaceBlockPlacementPolicy and AvailableSpaceRackFaultTolerantBlockPlacementPolicy is essentially identical.
therefore, a public class called AvailableSpaceBlockPlacementPolicyUtils has been created.
this enables any potential future optimizations to be made to AvailableSpaceBlockPlacementPolicyUtils, allowing for easy modifications to both policies.

Hi @ayushtkn @Hexiaoqiao @ZanderXu @tomscut @zhangshuyan0 Would you mind to take a review this pr? thank you very much~

hadoop-yetus · 2023-11-04T07:25:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 47s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 47s		trunk passed
+1 💚	mvnsite	0m 52s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 9s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 53s		trunk passed
+1 💚	shadedclient	21m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 35s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 228 unchanged - 6 fixed = 249 total (was 234)
+1 💚	mvnsite	0m 44s		the patch passed
+1 💚	javadoc	0m 36s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 50s		the patch passed
+1 💚	shadedclient	21m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	193m 7s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		285m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSUtil

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/2/artifact/out/Dockerfile
GITHUB PR	#6245
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux ab575bd87db0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5e99ff9`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/2/testReport/
Max. process+thread count	3526 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-11-04T13:32:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 49s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 44s		trunk passed
+1 💚	mvnsite	0m 52s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 8s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 58s		trunk passed
+1 💚	shadedclient	21m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 35s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 228 unchanged - 6 fixed = 249 total (was 234)
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 3s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 51s		the patch passed
+1 💚	shadedclient	21m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	192m 35s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		283m 46s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSUtil

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/3/artifact/out/Dockerfile
GITHUB PR	#6245
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux d84c2e9c7df1 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4ae1ed8`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/3/testReport/
Max. process+thread count	4060 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6245/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2023-12-02T09:27:30Z

Hi @ayushtkn Would you mind to take a review this pr when you have time? thank you very much~

Hexiaoqiao · 2023-12-03T11:41:51Z

Thanks involve me here and sorry to be late. I will try to review it next week. Thanks again.

HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPoli…

a988804

…cy logic

github-actions bot added HDFS trunk labels Nov 2, 2023

haiyang1987 changed the title ~~HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPoli…~~ HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic Nov 2, 2023

HDFS-17247. Fix some logic and checkstytle

5e99ff9

HDFS-17247. Fix checkstytle

4ae1ed8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic #6245

HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic #6245

haiyang1987 commented Nov 2, 2023 •

edited

Loading

hadoop-yetus commented Nov 2, 2023

haiyang1987 commented Nov 4, 2023

hadoop-yetus commented Nov 4, 2023

hadoop-yetus commented Nov 4, 2023

haiyang1987 commented Dec 2, 2023

Hexiaoqiao commented Dec 3, 2023

HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic #6245

Are you sure you want to change the base?

HDFS-17247. Improve AvailableSpaceRackFaultTolerantBlockPlacementPolicy logic #6245

Conversation

haiyang1987 commented Nov 2, 2023 • edited Loading

Description of PR

hadoop-yetus commented Nov 2, 2023

haiyang1987 commented Nov 4, 2023

hadoop-yetus commented Nov 4, 2023

hadoop-yetus commented Nov 4, 2023

haiyang1987 commented Dec 2, 2023

Hexiaoqiao commented Dec 3, 2023

haiyang1987 commented Nov 2, 2023 •

edited

Loading