HDFS-16445. Make HDFS count, mkdir, rm cross platform #3945

GauthamBanasandra · 2022-01-30T18:32:46Z

Description of PR

The source files for hdfs_count, hdfs_mkdir and hdfs_rm uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

This PR was tested by running the above tools locally.

hdfs_mkdir

Ran the tool to create a directory.

$ ./hdfs_mkdir /home/gautham/new-dir

The tool is then run with the -p option to exercise the creation of intermediate directories.

$ ./hdfs_mkdir -p /home/gautham/a/b/c

The trailing a/b/c directories are created under the path /home/gautham.

The tool is then run with the -m option with 700 as the mode. This turns all the permission bits on for user and turns off all the permission bits for group and other users.

$ ./hdfs_mkdir -p -m 700 /home/gautham/d/e/f

The trailing d/e/f directories are created under the path /home/gautham with the permission of 700, even for the intermediate directories.

Confirmation from browser.

hdfs_count

Ran the tool on one of the HDFS paths.

$ ./hdfs_count /home/gautham

It shows the count of files and directories contained within this path.

The tool is then run with the quota option.

$ ./hdfs_count -q /home/gautham

It displays the quota information additionally.

hdfs_rm

Ran the tool to delete a file.

$ ./hdfs_rm /home/gautham/data.txt

The tool was then run with the -R option to recursively delete a directory and its sub-directories.

$ ./hdfs_rm -R /home/gautham/a

The directory a and its sub-directories were deleted.

Confirmation from browser.

![Delete directory](https://user-images.githubusercontent.com/10280768/151838226-86f1fb90-b3ab-46dd

-b8ab-28c8cfd19c76.jpg)

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-01-30T22:18:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	43m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 55s		trunk passed
+1 💚	compile	3m 18s		trunk passed
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	shadedclient	60m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 18s		the patch passed
+1 💚	cc	3m 18s		the patch passed
+1 💚	golang	3m 18s		the patch passed
+1 💚	javac	3m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	21m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	92m 16s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		224m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/artifact/out/Dockerfile
GITHUB PR	#3945
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 517049edfaf0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `08bb31e`
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/testReport/
Max. process+thread count	529 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-31T01:46:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	26m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 19s		trunk passed
+1 💚	compile	3m 30s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	shadedclient	51m 53s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	3m 26s		the patch passed
+1 💚	cc	3m 26s		the patch passed
+1 💚	golang	3m 26s		the patch passed
+1 💚	javac	3m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	shadedclient	22m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	99m 59s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		207m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/artifact/out/Dockerfile
GITHUB PR	#3945
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux ffd8e2cd596b 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `08bb31e`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/testReport/
Max. process+thread count	519 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-31T05:26:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	15m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 30s		trunk passed
+1 💚	compile	3m 10s		trunk passed
+1 💚	mvnsite	0m 21s		trunk passed
+1 💚	shadedclient	66m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s		the patch passed
+1 💚	compile	3m 10s		the patch passed
+1 💚	cc	3m 10s		the patch passed
+1 💚	golang	3m 10s		the patch passed
+1 💚	javac	3m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	30m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	99m 38s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		219m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/artifact/out/Dockerfile
GITHUB PR	#3945
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 260bfc1e4d0a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `08bb31e`
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/testReport/
Max. process+thread count	586 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-01-31T08:41:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 2s		trunk passed
+1 💚	compile	3m 39s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	3m 44s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	shadedclient	54m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s		the patch passed
+1 💚	compile	3m 38s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	cc	3m 38s		the patch passed
+1 💚	golang	3m 38s		the patch passed
+1 💚	javac	3m 38s		the patch passed
+1 💚	compile	3m 42s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	3m 42s		the patch passed
+1 💚	golang	3m 42s		the patch passed
+1 💚	javac	3m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	22m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	106m 9s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		194m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/artifact/out/Dockerfile
GITHUB PR	#3945
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux e5365535cf36 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `08bb31e`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/testReport/
Max. process+thread count	597 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3945/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra added 15 commits January 30, 2022 23:45

Make HDFS count tool cross platform

d28df52

Add unit tests for HDFS count tool

ba54bcf

Fix unit tests for hdfs_count

ecde7d9

Remove redundant file hdfs_count.cc

db3543d

Make HDFS mkdir cross platform

62e32ea

Make GetPermissions protected

576235b

Add unit tests for HDFS mkdir tool

f8dd1f8

Fix unit tests

a6a1d12

Remove redundant file for HDFS mkdir tool

40bd539

Add TODO for HDFS mkdir tool

880dcfb

Make HDFS rm tool cross platform

51388f3

Rename the header constant for HDFS rm

e569f57

Add unit tests for HDFS rm tool

302f9af

Remove redundant file for HDFS rm tool

50927e9

Add documentation for HDFS mkdir tool

08bb31e

GauthamBanasandra marked this pull request as ready for review January 31, 2022 17:01

GauthamBanasandra self-assigned this Jan 31, 2022

GauthamBanasandra added the HDFS label Jan 31, 2022

goiri approved these changes Feb 1, 2022

View reviewed changes

GauthamBanasandra merged commit ed44662 into apache:trunk Feb 2, 2022

GauthamBanasandra deleted the count-mkdir-rm-x-platform branch February 2, 2022 05:42

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16445. Make HDFS count, mkdir, rm cross platform (apache#3945)

fa2c0a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16445. Make HDFS count, mkdir, rm cross platform #3945

HDFS-16445. Make HDFS count, mkdir, rm cross platform #3945

GauthamBanasandra commented Jan 30, 2022 •

edited

Loading

hadoop-yetus commented Jan 30, 2022

hadoop-yetus commented Jan 31, 2022

hadoop-yetus commented Jan 31, 2022

hadoop-yetus commented Jan 31, 2022

HDFS-16445. Make HDFS count, mkdir, rm cross platform #3945

HDFS-16445. Make HDFS count, mkdir, rm cross platform #3945

Conversation

GauthamBanasandra commented Jan 30, 2022 • edited Loading

Description of PR

How was this patch tested?

hdfs_mkdir

hdfs_count

hdfs_rm

For code changes:

hadoop-yetus commented Jan 30, 2022

hadoop-yetus commented Jan 31, 2022

hadoop-yetus commented Jan 31, 2022

hadoop-yetus commented Jan 31, 2022

GauthamBanasandra commented Jan 30, 2022 •

edited

Loading