HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background #3842

cndaimin · 2021-12-30T08:23:09Z

When we examining the FUSE IO performance on HDFS, we found that the simultaneous IO requests number are limited to a fixed number, like 12. This limitation makes the IO performance on FUSE client quite unacceptable. We did some research on this and inspired by the article Performance and Resource Utilization of FUSE User-Space File Systems, clearly the FUSE parameter max_background decides the simultaneous IO requests number, which is 12 by default.

We add max_background to fuse_dfs mount options, the FUSE kernel will take effect when an option value is given.

To check it works:

cat /sys/fs/fuse/connections/55/max_background
100

hadoop-yetus · 2021-12-30T11:59:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	39m 17s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 58s		trunk passed
+1 💚	compile	3m 4s		trunk passed
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed
+0 🆗	spotbugs	0m 28s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	19m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	2m 50s		the patch passed
+1 💚	cc	2m 50s		the patch passed
+1 💚	golang	2m 50s		the patch passed
+1 💚	javac	2m 50s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 5 line(s) with tabs.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed
+0 🆗	spotbugs	0m 16s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	18m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	92m 37s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		214m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/1/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 89e2f1e83293 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / aaa09b63b6873b46a0d8f1229f777a1b12be1feb
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/1/testReport/
Max. process+thread count	610 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-30T18:44:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	shellcheck	0m 1s		Shellcheck was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 34s		trunk passed
+1 💚	compile	19m 44s		trunk passed
+1 💚	checkstyle	3m 41s		trunk passed
+1 💚	mvnsite	2m 27s		trunk passed
+1 💚	javadoc	2m 31s		trunk passed
+0 🆗	spotbugs	0m 40s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	19m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	18m 57s		the patch passed
-1 ❌	cc	18m 57s	/results-compile-cc-root.txt	root generated 20 new + 185 unchanged - 20 fixed = 205 total (was 205)
+1 💚	golang	18m 57s		the patch passed
+1 💚	javac	18m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 37s		the patch passed
+1 💚	mvnsite	2m 26s		the patch passed
+1 💚	javadoc	2m 23s		the patch passed
+0 🆗	spotbugs	0m 40s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	19m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	17m 55s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	unit	93m 56s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		256m 32s

Reason	Tests
Failed junit tests	hadoop.ipc.TestIPC
	hadoop.crypto.TestCryptoCodec
	hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec
	hadoop.service.launcher.TestServiceInterruptHandling

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/2/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell shellcheck shelldocs cc golang
uname	Linux 9f07575dfb1b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3ae7929442781d4399e8bc9c8a3ec7a932a0ffd5
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/2/testReport/
Max. process+thread count	1281 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-native-client U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/2/console
versions	git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

sodonnel · 2022-01-04T11:42:57Z

There are some changes here related to erasure coding which don't seem relevant to this Jira. Have they been included by mistake?

cndaimin · 2022-01-05T05:30:13Z

There are some changes here related to erasure coding which don't seem relevant to this Jira. Have they been included by mistake?

Sorry for that, I will revert this part.

hadoop-yetus · 2022-01-05T09:00:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 4s		trunk passed
+1 💚	compile	3m 10s		trunk passed
+1 💚	checkstyle	0m 25s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed
+0 🆗	spotbugs	0m 29s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	19m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 51s		the patch passed
+1 💚	cc	2m 51s		the patch passed
+1 💚	golang	2m 51s		the patch passed
+1 💚	javac	2m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed
+0 🆗	spotbugs	0m 16s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	18m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	104m 43s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		188m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 16cf458787de 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/testReport/
Max. process+thread count	548 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-05T09:30:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	43m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 57s		trunk passed
+1 💚	compile	3m 13s		trunk passed
+1 💚	checkstyle	0m 18s		trunk passed
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	javadoc	0m 21s		trunk passed
+0 🆗	spotbugs	0m 22s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	21m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 5s		the patch passed
+1 💚	cc	3m 5s		the patch passed
+1 💚	golang	3m 5s		the patch passed
+1 💚	javac	3m 5s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 10s		the patch passed
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	javadoc	0m 12s		the patch passed
+0 🆗	spotbugs	0m 13s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	21m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	82m 55s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		217m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 7eeaf232f40d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/testReport/
Max. process+thread count	630 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-05T12:36:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	30m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 25s		trunk passed
+1 💚	compile	3m 50s		trunk passed
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 35s		trunk passed
+1 💚	javadoc	0m 31s		trunk passed
+0 🆗	spotbugs	0m 32s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	22m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	4m 0s		the patch passed
+1 💚	cc	4m 0s		the patch passed
+1 💚	golang	4m 0s		the patch passed
+1 💚	javac	4m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed
+0 🆗	spotbugs	0m 18s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	23m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	67m 50s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		185m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 6054726b5b4d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/testReport/
Max. process+thread count	576 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-05T13:02:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	27m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 50s		trunk passed
+1 💚	compile	3m 22s		trunk passed
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed
+0 🆗	spotbugs	0m 43s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	21m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 24s		the patch passed
+1 💚	cc	3m 24s		the patch passed
+1 💚	golang	3m 24s		the patch passed
+1 💚	javac	3m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed
+0 🆗	spotbugs	0m 23s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	22m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	128m 52s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		241m 47s

Reason	Tests
Failed CTEST tests	test_test_libhdfs_threaded_hdfs_static
	test_test_libhdfs_zerocopy_hdfs_static
	test_test_native_mini_dfs
	remote_block_reader
	memcheck_remote_block_reader
	sasl_digest_md5
	memcheck_sasl_digest_md5
	retry_policy
	memcheck_retry_policy
	rpc_engine
	memcheck_rpc_engine
	bad_datanode
	memcheck_bad_datanode
	node_exclusion
	memcheck_node_exclusion
	configuration
	memcheck_configuration
	hdfs_configuration
	memcheck_hdfs_configuration
	hdfspp_errors
	memcheck_hdfspp_errors
	hdfs_builder_test
	memcheck_hdfs_builder_test
	logging_test
	memcheck_logging_test
	hdfs_ioservice
	memcheck_hdfs_ioservice
	user_lock
	memcheck_user_lock
	hdfs_config_connect_bugs
	memcheck_hdfs_config_connect_bugs
	test_libhdfs_threaded_hdfspp_test_shim_static
	test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
	libhdfs_mini_stress_valgrind_hdfspp_test_static
	memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
	test_libhdfs_mini_stress_hdfspp_test_shim_static
	test_hdfs_ext_hdfspp_test_shim_static
	x_platform_utils_test
	x_platform_syscall_test
	hdfs_tool_tests

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 6549990b31ad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Red Hat, Inc.-1.8.0_312-b07
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/testReport/
Max. process+thread count	682 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/4/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-05T15:45:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	21m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 4s		trunk passed
+1 💚	compile	3m 21s		trunk passed
+1 💚	checkstyle	0m 19s		trunk passed
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	javadoc	0m 21s		trunk passed
+0 🆗	spotbugs	0m 22s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	31m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	3m 3s		the patch passed
+1 💚	cc	3m 3s		the patch passed
+1 💚	golang	3m 3s		the patch passed
+1 💚	javac	3m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	0m 13s		the patch passed
+0 🆗	spotbugs	0m 16s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	30m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	58m 51s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		189m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 5c26a2bdf7e1 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Debian-11.0.13+8-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/testReport/
Max. process+thread count	610 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/console
versions	git=2.20.1 maven=3.6.0 shellcheck=0.5.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-05T18:32:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 37s		trunk passed
+1 💚	compile	3m 32s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	3m 28s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 16s		trunk passed
+1 💚	mvnsite	0m 20s		trunk passed
+1 💚	javadoc	0m 18s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 18s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 19s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	22m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 24s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	3m 24s		the patch passed
+1 💚	golang	3m 24s		the patch passed
+1 💚	javac	3m 24s		the patch passed
+1 💚	compile	3m 27s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	3m 27s		the patch passed
+1 💚	golang	3m 27s		the patch passed
+1 💚	javac	3m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 10s		the patch passed
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	0m 13s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 13s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 14s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	22m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	76m 21s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		166m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux 8f72fd1d301e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae8786d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/testReport/
Max. process+thread count	519 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

cndaimin · 2022-01-06T03:11:04Z

@sodonnel I have removed the unrelated code, could you please take a look again? Thanks.

jojochuang · 2022-01-12T03:18:15Z

@fapifta do you have some cycles to review this one? This is a great addition.

fapifta · 2022-01-13T17:50:40Z

@jojochuang I did not have the time so far to thoroughly go through the linked paper, or test out the new code, but for the first sight, the option seems to be reasonable, and it seems to be added properly. For more extensive testing, I might have some time next week, but so far I can not promise it. Overall, the finding looks promising, and the implementation itself seems to be seamless with the related part of the code, so I am brave enough to say that I am fine with committing it.

@cndaimin Can you share some details about your test results? I guess you have some measures with the implementation, is it something you can share? I would be curious what gains you can get with more threads, and if and where the gain is getting less and less when one adds more background threads.

cndaimin · 2022-01-14T07:09:00Z

@jojochuang @fapifta Thanks for the review.

We use fio with 60 threads to do random read on files under test directory and to measure the performance by read IOPS, test steps as following:

Prepare test files to speed up the next random reads.
Drop page cache of both client and datanode servers.
Do the random read test with 60 threads.

The test scripts:

# Prepare test files to speed up the next random reads.
fio -iodepth=32 -rw=write -ioengine=libaio -bs=4096k -size=1G -direct=0 -runtime=600 -directory=/mnt/dfs/iotest -numjobs=60 -thread -group_reporting -name=i

# Drop page cache of both client and datanode servers.
pssh -h ~/hosts -t 0 -i "sync && echo 1 > /proc/sys/vm/drop_caches"

# Do the random read test.
fio -iodepth=1 -rw=randread -ioengine=libaio -bs=512 -size=1G -direct=0 -runtime=120 -directory=/mnt/dfs/iotest -numjobs=60 -thread -group_reporting -name=i

And the test results:

With default max_background, which is 12

# cat /sys/fs/fuse/connections/55/max_background
12
# fio -iodepth=1 -rw=randread -ioengine=libaio -bs=512 -size=1G -direct=0 -runtime=120 -directory=/mnt/dfs/iotest -numjobs=60 -thread -group_reporting -name=i
i: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 60 threads
Jobs: 60 (f=60): [r(60)][100.0%][r=722KiB/s,w=0KiB/s][r=1444,w=0 IOPS][eta 00m:00s]]
i: (groupid=0, jobs=60): err= 0: pid=13143: Fri Jan 14 14:34:35 2022
   read: IOPS=1365, BW=683KiB/s (699kB/s)(80.0MiB/120043msec)
    slat (nsec): min=1868, max=331615k, avg=43862093.94, stdev=10900971.93
    clat (nsec): min=615, max=234495, avg=2124.79, stdev=1111.62
     lat (usec): min=2, max=331618, avg=43865.27, stdev=10901.06
    clat percentiles (nsec):
     |  1.00th=[ 1176],  5.00th=[ 1416], 10.00th=[ 1528], 20.00th=[ 1704],
     | 30.00th=[ 1864], 40.00th=[ 1992], 50.00th=[ 2064], 60.00th=[ 2160],
     | 70.00th=[ 2256], 80.00th=[ 2384], 90.00th=[ 2576], 95.00th=[ 2768],
     | 99.00th=[ 3408], 99.50th=[ 7968], 99.90th=[16064], 99.95th=[19584],
     | 99.99th=[27008]
   bw (  KiB/s): min=    0, max=   16, per=1.66%, avg=11.34, stdev= 1.25, samples=14398
   iops        : min=    1, max=   32, avg=22.72, stdev= 2.48, samples=14398
  lat (nsec)   : 750=0.34%, 1000=0.10%
  lat (usec)   : 2=40.90%, 4=57.85%, 10=0.51%, 20=0.25%, 50=0.05%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.02%, sys=0.04%, ctx=324069, majf=0, minf=60
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=163917,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=683KiB/s (699kB/s), 683KiB/s-683KiB/s (699kB/s-699kB/s), io=80.0MiB (83.9MB), run=120043-120043msec

With -omax_background=100

# cat /sys/fs/fuse/connections/55/max_background
100
# fio -iodepth=1 -rw=randread -ioengine=libaio -bs=512 -size=1G -direct=0 -runtime=120 -directory=/mnt/dfs/iotest -numjobs=60 -thread -group_reporting -name=i
i: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 60 threads
# Check JAVA_HOME
Jobs: 60 (f=60): [r(60)][100.0%][r=1768KiB/s,w=0KiB/s][r=3536,w=0 IOPS][eta 00m:00s]]
i: (groupid=0, jobs=60): err= 0: pid=12582: Fri Jan 14 14:31:25 2022
   read: IOPS=3569, BW=1785KiB/s (1828kB/s)(209MiB/120037msec)
    slat (nsec): min=1576, max=708718k, avg=16797865.36, stdev=16920653.78
    clat (nsec): min=603, max=343984, avg=2023.56, stdev=1829.69
     lat (usec): min=2, max=708721, avg=16800.82, stdev=16920.78
    clat percentiles (nsec):
     |  1.00th=[  748],  5.00th=[ 1224], 10.00th=[ 1400], 20.00th=[ 1592],
     | 30.00th=[ 1736], 40.00th=[ 1848], 50.00th=[ 1928], 60.00th=[ 2008],
     | 70.00th=[ 2096], 80.00th=[ 2192], 90.00th=[ 2352], 95.00th=[ 2512],
     | 99.00th=[ 7648], 99.50th=[11968], 99.90th=[21632], 99.95th=[25984],
     | 99.99th=[39168]
   bw (  KiB/s): min=    6, max=   54, per=1.67%, avg=29.71, stdev= 6.19, samples=14397
   iops        : min=   12, max=  108, avg=59.46, stdev=12.37, samples=14397
  lat (nsec)   : 750=0.99%, 1000=0.80%
  lat (usec)   : 2=57.08%, 4=39.87%, 10=0.60%, 20=0.53%, 50=0.13%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=0.04%, sys=0.08%, ctx=423468, majf=0, minf=60
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=428483,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1785KiB/s (1828kB/s), 1785KiB/s-1785KiB/s (1828kB/s-1828kB/s), io=209MiB (219MB), run=120037-120037msec

In our test, by setting max_background to 100 will improve the read IOPS from 1365 to 3569. And when the resources like cpu/memory are sufficient(which is not a problem generally), there seems no side effects of setting a bigger value. We have been running -omax_background=100 in our production environment for months and it looks good. @fapifta

fapifta · 2022-01-24T23:40:18Z

Hi @cndaimin sorry for the long reply time, I could not get to this one last week, and still I was not able to review the linked study but I am pretty much convinced by the numbers. I don't see problems with the patch either, so form my side I can give my non-binding +1 for commit. I still would like to run some tests, but based on my current schedule I think we should not block this until that point when I can.
@jojochuang if you are comfortable to +1 based on just this, and the numbers/experience shared about the patch in action, please do so ;)

jojochuang

The performance number looks great. I am +1. If the indentation in fuse_option.c and fuse_option.h can be updated, that would be even better.

Otherwise I will merge by tomorrow.

cndaimin · 2022-01-25T04:25:24Z

The indentation difference seems to be gap between tab and space, I just fixed it. Thanks a lot. @jojochuang

jojochuang

Great. +1 merging it now.

hadoop-yetus · 2022-01-25T07:57:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	38m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 11s		trunk passed
+1 💚	compile	3m 5s		trunk passed
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed
+0 🆗	spotbugs	0m 29s		branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	18m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	2m 55s		the patch passed
+1 💚	cc	2m 55s		the patch passed
+1 💚	golang	2m 55s		the patch passed
+1 💚	javac	2m 55s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed
+0 🆗	spotbugs	0m 16s		hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚	shadedclient	18m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	95m 40s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		217m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/5/artifact/out/Dockerfile
GITHUB PR	#3842
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs checkstyle
uname	Linux e87d93654e61 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d44d429`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/5/testReport/
Max. process+thread count	715 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/5/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-25T07:59:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 19s		#3842 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#3842
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

…max_background (#3842) Reviewed-by: Istvan Fajth <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit d699389)

…max_background (#3842) Reviewed-by: Istvan Fajth <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]> (cherry picked from commit d699389) (cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)

…max_background (apache#3842) Reviewed-by: Istvan Fajth <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]>

cndaimin force-pushed the max_background branch from 3ae7929 to a2b5cde Compare January 5, 2022 05:51

cndaimin added 2 commits January 5, 2022 13:51

Improve FUSE IO performance by supporting FUSE parameter max_background

e49762e

Fix checkstyle

ae8786d

cndaimin force-pushed the max_background branch from a2b5cde to ae8786d Compare January 5, 2022 05:51

jojochuang reviewed Jan 25, 2022

View reviewed changes

Fix code indentation

d44d429

jojochuang approved these changes Jan 25, 2022

View reviewed changes

jojochuang merged commit d699389 into apache:trunk Jan 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background #3842

HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background #3842

cndaimin commented Dec 30, 2021 •

edited

Loading

hadoop-yetus commented Dec 30, 2021

hadoop-yetus commented Dec 30, 2021

sodonnel commented Jan 4, 2022

cndaimin commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

cndaimin commented Jan 6, 2022

jojochuang commented Jan 12, 2022

fapifta commented Jan 13, 2022

cndaimin commented Jan 14, 2022 •

edited

Loading

fapifta commented Jan 24, 2022

jojochuang left a comment

cndaimin commented Jan 25, 2022

jojochuang left a comment

hadoop-yetus commented Jan 25, 2022

hadoop-yetus commented Jan 25, 2022

HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background #3842

HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background #3842

Conversation

cndaimin commented Dec 30, 2021 • edited Loading

hadoop-yetus commented Dec 30, 2021

hadoop-yetus commented Dec 30, 2021

sodonnel commented Jan 4, 2022

cndaimin commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

hadoop-yetus commented Jan 5, 2022

cndaimin commented Jan 6, 2022

jojochuang commented Jan 12, 2022

fapifta commented Jan 13, 2022

cndaimin commented Jan 14, 2022 • edited Loading

fapifta commented Jan 24, 2022

jojochuang left a comment

Choose a reason for hiding this comment

cndaimin commented Jan 25, 2022

jojochuang left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Jan 25, 2022

hadoop-yetus commented Jan 25, 2022

cndaimin commented Dec 30, 2021 •

edited

Loading

cndaimin commented Jan 14, 2022 •

edited

Loading