HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options #2751

lamberken · 2021-03-07T08:02:47Z

ISSUE

https://issues.apache.org/jira/browse/HDFS-15882

NOTICE

rw Open for reading and writing. If the file does not already exist then an attempt will be made to create it.
rws Require that every update to the file's content or metadata be written synchronously to the underlying storage device.

From the literal meaning of this variable shouldSyncWritesAndSkipFsync, we should use rw when shouldSyncWritesAndSkipFsync is true.

We use SATA disk to store the journal node's data. It's not effective for improving RPC performance whether the shouldSyncWritesAndSkipFsync variable is true or false. it's caused by initializing RandomAccessFile incorrectly.

lamberken · 2021-03-07T08:04:12Z

hi @aajisaka, please take a review when you're free, thanks.

Hexiaoqiao

LGTM. +1 from my side. Will commit if no other comments shortly.

leosunli · 2021-03-07T10:49:41Z

LGTM

lamberken · 2021-03-07T11:12:44Z

Thanks @Hexiaoqiao @leosunli

lamberken · 2021-03-07T16:24:27Z

I think the test failures are not related to the patch, and the failed tests succeeded on my local computer.

lamberken · 2021-03-10T13:14:55Z

After patching, the performance is nearly doubled.

Hexiaoqiao · 2021-03-10T13:37:06Z

Sorry for late response, just check yetus report, I found there are many failed unit tests. Do you mind to trigger Yetus again?
BTW, application execution time is not the proper measure for this improvement IMO, just suggest to use nnbench or similar tools to check it. Thanks.

lamberken · 2021-03-10T14:45:23Z

use nnbench or similar tools to check it. Thanks.

Yeah, we use nnbench tool to test create_write.

hadoop dfs -rm -r /tmp/nnbench
hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.0-tests.jar nnbench \
-Ddfs.checksum.type=NULL \
-operation create_write -maps 200 -reduces 15 \
-bytesToWrite 1 -numberOfFiles 5000 \
-blockSize 1048576 -replicationFactorPerFile 3 \
-readFileAfterOpen true -baseDir /tmp/nnbench

…configuration options

hadoop-yetus · 2021-03-12T16:25:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 5s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 28s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 24s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 29s		trunk passed
+1 💚	shadedclient	19m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 17s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 33s		the patch passed
+1 💚	shadedclient	19m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	341m 26s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		440m 24s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.tools.TestDFSZKFailoverController
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestHDFSFileSystemContract
	hadoop.hdfs.TestPersistBlocks
	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/6/artifact/out/Dockerfile
GITHUB PR	#2751
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 828f0a5da9fd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4c2aa66b8400d1f448b03a52430fef9abe656921
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/6/testReport/
Max. process+thread count	2092 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-12T16:32:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 50s		trunk passed
+1 💚	compile	1m 38s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 27s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 35s		trunk passed
+1 💚	javadoc	1m 6s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	4m 7s		trunk passed
+1 💚	shadedclient	20m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 25s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 34s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 56s		the patch passed
+1 💚	shadedclient	21m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	237m 35s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		341m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/7/artifact/out/Dockerfile
GITHUB PR	#2751
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 6dca0d31689e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d18a66f`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/7/testReport/
Max. process+thread count	3047 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2751/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2021-03-13T06:23:04Z

The failed unit tests seems not related to this PR, committed to trunk. Thanks @lamberken for your report and contributions.

…nfiguration options (apache#2751). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <[email protected]>

Hexiaoqiao changed the title ~~HADOOP-15882. Fix incorrectly initializing RandomAccessFile based on configuration options~~ HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options Mar 7, 2021

Hexiaoqiao approved these changes Mar 7, 2021

View reviewed changes

lamberken force-pushed the patch-15882 branch 2 times, most recently from 976fa03 to d380f26 Compare March 11, 2021 15:35

HADOOP-15882. Fix incorrectly initializing RandomAccessFile based on …

d18a66f

…configuration options

lamberken force-pushed the patch-15882 branch from 4c2aa66 to d18a66f Compare March 12, 2021 10:49

Hexiaoqiao merged commit 6921ec8 into apache:trunk Mar 13, 2021

lamberken deleted the patch-15882 branch May 26, 2021 14:02

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15882. Fix incorrectly initializing RandomAccessFile based on co…

b5ffc0d

…nfiguration options (apache#2751). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options #2751

HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options #2751

lamberken commented Mar 7, 2021 •

edited

Loading

lamberken commented Mar 7, 2021

Hexiaoqiao left a comment

leosunli commented Mar 7, 2021

lamberken commented Mar 7, 2021

lamberken commented Mar 7, 2021 •

edited

Loading

lamberken commented Mar 10, 2021 •

edited

Loading

Hexiaoqiao commented Mar 10, 2021

lamberken commented Mar 10, 2021 •

edited

Loading

hadoop-yetus commented Mar 12, 2021

hadoop-yetus commented Mar 12, 2021

Hexiaoqiao commented Mar 13, 2021

HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options #2751

HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options #2751

Conversation

lamberken commented Mar 7, 2021 • edited Loading

ISSUE

NOTICE

lamberken commented Mar 7, 2021

Hexiaoqiao left a comment

Choose a reason for hiding this comment

leosunli commented Mar 7, 2021

lamberken commented Mar 7, 2021

lamberken commented Mar 7, 2021 • edited Loading

lamberken commented Mar 10, 2021 • edited Loading

Hexiaoqiao commented Mar 10, 2021

lamberken commented Mar 10, 2021 • edited Loading

hadoop-yetus commented Mar 12, 2021

hadoop-yetus commented Mar 12, 2021

Hexiaoqiao commented Mar 13, 2021

lamberken commented Mar 7, 2021 •

edited

Loading

lamberken commented Mar 7, 2021 •

edited

Loading

lamberken commented Mar 10, 2021 •

edited

Loading

lamberken commented Mar 10, 2021 •

edited

Loading