HDFS-16472. Make HDFS setrep tool cross platform #4130

GauthamBanasandra · 2022-04-01T18:22:45Z

Description of PR

The source files for hdfs_setrep uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make this tool cross platform.

How was this patch tested?

hdfs_setrep

This tool was tested by running against a Hadoop cluster setup in pseduo-distributed mode. The details are as follows -
There's a file initially in the HDFS path ~/some-dir/some-data.txt that has an initial replication factor of 1.

We change it to 4 by running the following command.

$ ./hdfs_setrep 4 ~/some-dir/some-data.txt

Confirmation from UI.

If the path provided to hdfs_setrep happens to be a directory, it recursively sets the replication factor to all its children. Here, we set the replication factor to 6 from the root.

$ ./hdfs_setrep 6 /

Confirmation from UI.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-04-01T22:36:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	45m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	37m 22s		trunk passed
+1 💚	compile	3m 40s		trunk passed
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	shadedclient	64m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 13s		the patch passed
+1 💚	compile	3m 33s		the patch passed
+1 💚	cc	3m 33s		the patch passed
+1 💚	golang	3m 33s		the patch passed
+1 💚	javac	3m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	22m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	114m 16s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		252m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/artifact/out/Dockerfile
GITHUB PR	#4130
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 29c424cb8230 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eeb5ad0`
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/testReport/
Max. process+thread count	599 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-02T02:27:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	26m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	26m 12s		trunk passed
+1 💚	compile	3m 49s		trunk passed
+1 💚	mvnsite	0m 36s		trunk passed
+1 💚	shadedclient	53m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	3m 36s		the patch passed
+1 💚	cc	3m 36s		the patch passed
+1 💚	golang	3m 36s		the patch passed
+1 💚	javac	3m 36s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	shadedclient	22m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	120m 14s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		230m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/artifact/out/Dockerfile
GITHUB PR	#4130
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 1217d4ef97ee 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eeb5ad0`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/testReport/
Max. process+thread count	597 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-02T06:22:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 45s		trunk passed
+1 💚	compile	3m 27s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	68m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 27s		the patch passed
+1 💚	cc	3m 27s		the patch passed
+1 💚	golang	3m 27s		the patch passed
+1 💚	javac	3m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	31m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	111m 46s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		234m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/artifact/out/Dockerfile
GITHUB PR	#4130
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 1fbd5b2ab325 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eeb5ad0`
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/testReport/
Max. process+thread count	570 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-02T09:57:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 46s		trunk passed
+1 💚	compile	4m 1s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	4m 4s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 24s		trunk passed
+1 💚	shadedclient	56m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 59s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	3m 59s		the patch passed
+1 💚	golang	3m 59s		the patch passed
+1 💚	javac	3m 59s		the patch passed
+1 💚	compile	4m 12s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	4m 12s		the patch passed
+1 💚	golang	4m 12s		the patch passed
+1 💚	javac	4m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	23m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	122m 37s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		214m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/artifact/out/Dockerfile
GITHUB PR	#4130
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 3bd1b8383bb2 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eeb5ad0`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/testReport/
Max. process+thread count	520 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4130/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* The source files for hdfs_setrep uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make this tool cross platform.

GauthamBanasandra added 7 commits March 29, 2022 21:00

Make HDFS setrep cross platform

73704fb

Add files for unit test

ba7698e

Add replication-factor param

f645eec

Add unit tests

fcc124c

Add documentation

fe5f947

Remove redundant file

d804cad

Add newline to set-replication-state.h

eeb5ad0

GauthamBanasandra self-assigned this Apr 1, 2022

GauthamBanasandra added the HDFS label Apr 1, 2022

GauthamBanasandra marked this pull request as ready for review April 2, 2022 10:09

GauthamBanasandra requested a review from goiri April 2, 2022 10:09

goiri approved these changes Apr 4, 2022

View reviewed changes

GauthamBanasandra merged commit 4ef1d3e into apache:trunk Apr 5, 2022

GauthamBanasandra deleted the setrep-x-platform branch April 5, 2022 17:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16472. Make HDFS setrep tool cross platform #4130

HDFS-16472. Make HDFS setrep tool cross platform #4130

GauthamBanasandra commented Apr 1, 2022 •

edited

Loading

hadoop-yetus commented Apr 1, 2022

hadoop-yetus commented Apr 2, 2022

hadoop-yetus commented Apr 2, 2022

hadoop-yetus commented Apr 2, 2022

HDFS-16472. Make HDFS setrep tool cross platform #4130

HDFS-16472. Make HDFS setrep tool cross platform #4130

Conversation

GauthamBanasandra commented Apr 1, 2022 • edited Loading

Description of PR

How was this patch tested?

hdfs_setrep

For code changes:

hadoop-yetus commented Apr 1, 2022

hadoop-yetus commented Apr 2, 2022

hadoop-yetus commented Apr 2, 2022

hadoop-yetus commented Apr 2, 2022

GauthamBanasandra commented Apr 1, 2022 •

edited

Loading