HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. #4091

omalley · 2022-03-22T17:04:27Z

Description of PR

The distance metric used for machines in 2.10 that aren't in the NetworkTopology, because they aren't running DataNodes, is wrong. It means that off-rack and on-rack, but off-node, are both given a weight of 2. In normal Hadoop clusters, this isn't a big problem because they don't have clients that are on-rack but without DataNodes. For clusters that are striped (federated HDFS going across racks) or separate compute and storage that share racks are both really bad with this bug.

How was this patch tested?

Unit test added.

hadoop-yetus · 2022-03-22T19:01:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-2.10 Compile Tests _
+1 💚	mvninstall	14m 47s		branch-2.10 passed
+1 💚	compile	13m 52s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	11m 25s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	0m 43s		branch-2.10 passed
+1 💚	mvnsite	1m 20s		branch-2.10 passed
+1 💚	javadoc	1m 16s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 7s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 6s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 45s		the patch passed
+1 💚	compile	13m 8s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	13m 8s		the patch passed
+1 💚	compile	11m 16s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	11m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	1m 16s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 4s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	2m 9s		the patch passed
			_ Other Tests _
-1 ❌	unit	9m 20s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		101m 36s

Reason	Tests
Failed junit tests	hadoop.fs.sftp.TestSFTPFileSystem
	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.io.compress.TestCompressorDecompressor

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/Dockerfile
GITHUB PR	#4091
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 6317914f6ae6 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / be14b0cad50fed2d8ecb42d236aed397bebc9609
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/testReport/
Max. process+thread count	1397 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-22T19:26:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-2.10 Compile Tests _
+1 💚	mvninstall	15m 55s		branch-2.10 passed
+1 💚	compile	14m 15s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	13m 9s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	0m 36s		branch-2.10 passed
+1 💚	mvnsite	1m 25s		branch-2.10 passed
+1 💚	javadoc	1m 17s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 0s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 4s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	13m 52s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
-1 ❌	javac	13m 52s	/results-compile-javac-root-jdkAzulSystems,Inc.-1.7.0_262-b10.txt	root-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 1678 unchanged - 1 fixed = 1679 total (was 1679)
+1 💚	compile	11m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	javac	11m 35s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt	root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~18.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~18.04-b07 generated 1 new + 1578 unchanged - 1 fixed = 1579 total (was 1579)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 35s		the patch passed
+1 💚	mvnsite	1m 17s		the patch passed
+1 💚	javadoc	1m 12s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	0m 59s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	2m 20s		the patch passed
			_ Other Tests _
-1 ❌	unit	10m 29s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		110m 7s

Reason	Tests
Failed junit tests	hadoop.io.compress.TestCompressorDecompressor
	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.fs.sftp.TestSFTPFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/Dockerfile
GITHUB PR	#4091
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux a08702ad2c29 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / 1d7220b772d64c1e9c40276e37738a0d0021095b
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/testReport/
Max. process+thread count	1534 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-22T22:56:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-2.10 Compile Tests _
+1 💚	mvninstall	14m 11s		branch-2.10 passed
+1 💚	compile	12m 56s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	10m 44s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	0m 42s		branch-2.10 passed
+1 💚	mvnsite	1m 19s		branch-2.10 passed
+1 💚	javadoc	1m 16s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 4s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 1s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 44s		the patch passed
+1 💚	compile	12m 21s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	12m 21s		the patch passed
+1 💚	compile	10m 37s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	10m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	javadoc	1m 15s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 2s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	2m 10s		the patch passed
			_ Other Tests _
-1 ❌	unit	9m 12s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		89m 53s

Reason	Tests
Failed junit tests	hadoop.fs.sftp.TestSFTPFileSystem
	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.util.TestDiskCheckerWithDiskIo
	hadoop.io.compress.TestCompressorDecompressor

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/Dockerfile
GITHUB PR	#4091
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 6b99d20a8120 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / 1326ef2a4f0d22c365603144812fac73dfa1f962
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/testReport/
Max. process+thread count	1393 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java

hadoop-yetus · 2022-03-23T18:47:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-2.10 Compile Tests _
+1 💚	mvninstall	15m 27s		branch-2.10 passed
+1 💚	compile	15m 36s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	11m 49s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	0m 38s		branch-2.10 passed
+1 💚	mvnsite	1m 22s		branch-2.10 passed
+1 💚	javadoc	1m 21s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 5s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 11s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 47s		the patch passed
+1 💚	compile	14m 24s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	14m 24s		the patch passed
+1 💚	compile	12m 15s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	12m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	1m 22s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	1m 7s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	2m 18s		the patch passed
			_ Other Tests _
-1 ❌	unit	9m 39s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 51s		The patch does not generate ASF License warnings.
		100m 9s

Reason	Tests
Failed junit tests	hadoop.fs.sftp.TestSFTPFileSystem
	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.util.TestBasicDiskValidator
	hadoop.io.compress.TestCompressorDecompressor

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/4/artifact/out/Dockerfile
GITHUB PR	#4091
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4ecc0fa7fadb 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / c4db7feb4fe29a16eaaea907fafcf8f965bffb32
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/4/testReport/
Max. process+thread count	1390 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

omalley · 2022-03-23T21:47:52Z

When I created the PR, I hadn't found the upstream jira, which is https://issues.apache.org/jira/browse/HADOOP-16161 .

…port of HADOOP-16161. Fixes apache#4091 Signed-off-by: Owen O'Malley <[email protected]>

hadoop-yetus · 2022-03-23T21:54:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 19s		#4091 does not apply to branch-2.10. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#4091
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

…d in HADOOP-16161, but this test case adds value to ensure the two getWeight methods stay in sync. Fixes #4091 Signed-off-by: Owen O'Malley <[email protected]>

…d in HADOOP-16161, but this test case adds value to ensure the two getWeight methods stay in sync. Fixes apache#4091 Signed-off-by: Owen O'Malley <[email protected]>

omalley force-pushed the hdfs-16517-2.10 branch from 1d7220b to 1326ef2 Compare March 22, 2022 21:25

xinglin suggested changes Mar 23, 2022

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java Outdated Show resolved Hide resolved

goiri approved these changes Mar 23, 2022

View reviewed changes

HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Back…

f1c3018

…port of HADOOP-16161. Fixes apache#4091 Signed-off-by: Owen O'Malley <[email protected]>

omalley force-pushed the hdfs-16517-2.10 branch from c4db7fe to f1c3018 Compare March 23, 2022 21:51

omalley merged commit f1c3018 into apache:branch-2.10 Mar 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. #4091

HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. #4091

omalley commented Mar 22, 2022

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 23, 2022

omalley commented Mar 23, 2022

hadoop-yetus commented Mar 23, 2022

HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. #4091

HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. #4091

Conversation

omalley commented Mar 22, 2022

Description of PR

How was this patch tested?

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 22, 2022

hadoop-yetus commented Mar 23, 2022

omalley commented Mar 23, 2022

hadoop-yetus commented Mar 23, 2022