HDFS-16564. Use uint32_t for hdfs_find #4245

GauthamBanasandra · 2022-04-28T17:07:31Z

Description of PR

hdfs_find uses u_int32_t type for storing the value for the max-depth command line argument -

hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-find/hdfs-find.cc

Line 43 in a631f45

"max-depth,m", po::value<u_int32_t>(),

.
The type u_int32_t isn't standard, isn't available on Windows and thus breaks cross-platform compatibility. We need to replace this with uint32_t which is available on all platforms since it's part of the C++ standard.

How was this patch tested?

The existing unit tests exercise this PR sufficiently.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

...s-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs_smoke.cc

steveloughran · 2022-04-30T11:12:24Z

this going to cause any linkage problems with existing apps?
what's up with the shaded client failure? its probably completely unrelated, but I've not seen this before.

GauthamBanasandra · 2022-04-30T14:23:30Z

Hi @steveloughran,

this going to cause any linkage problems with existing apps?

No. The type uint32_t is part of the C++ standard included in the cstdint header file. It shouldn't cause any issues.

what's up with the shaded client failure? its probably completely unrelated, but I've not seen this before.

Yeah. It seems like we've a regression. I was able to reproduce this on the trunk on Debian 10. I'm trying to do a git bisect to narrow down the commit that's causing this.
Here's the test failure -

[INFO] Running org.apache.hadoop.example.ITUseMiniCluster
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 18.47 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
[ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster)  Time elapsed: 12.167 s  <<< ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
	at org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
	at org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
	at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
	at org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

* Need to use uint32_t since it's available on all platform as it's part of the C++ standard.

hadoop-yetus · 2022-05-03T06:05:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 3s		trunk passed
+1 💚	compile	4m 16s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	shadedclient	69m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 54s		the patch passed
+1 💚	cc	3m 54s		the patch passed
+1 💚	golang	3m 54s		the patch passed
+1 💚	javac	3m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 27s		the patch passed
+1 💚	shadedclient	20m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	37m 34s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		136m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/artifact/out/Dockerfile
GITHUB PR	#4245
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux a178eced3645 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eccc29e`
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/testReport/
Max. process+thread count	685 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-03T07:56:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 24s		trunk passed
+1 💚	compile	4m 8s		trunk passed
+1 💚	mvnsite	1m 3s		trunk passed
+1 💚	shadedclient	47m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	3m 42s		the patch passed
+1 💚	cc	3m 42s		the patch passed
+1 💚	golang	3m 42s		the patch passed
+1 💚	javac	3m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	shadedclient	19m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	33m 45s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		110m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/artifact/out/Dockerfile
GITHUB PR	#4245
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 20effa88d310 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eccc29e`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/testReport/
Max. process+thread count	570 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-03T10:00:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 19s		trunk passed
+1 💚	compile	3m 39s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	shadedclient	57m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 21s		the patch passed
+1 💚	cc	3m 21s		the patch passed
+1 💚	golang	3m 21s		the patch passed
+1 💚	javac	3m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 27s		the patch passed
+1 💚	shadedclient	26m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 28s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		124m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/artifact/out/Dockerfile
GITHUB PR	#4245
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux fb52732e05c2 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eccc29e`
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/testReport/
Max. process+thread count	695 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-03T11:56:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 29s		trunk passed
+1 💚	compile	4m 2s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	4m 5s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	50m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 43s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	cc	3m 43s		the patch passed
+1 💚	golang	3m 43s		the patch passed
+1 💚	javac	3m 43s		the patch passed
+1 💚	compile	3m 46s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	3m 46s		the patch passed
+1 💚	golang	3m 46s		the patch passed
+1 💚	javac	3m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 28s		the patch passed
+1 💚	shadedclient	19m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	33m 36s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		115m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/artifact/out/Dockerfile
GITHUB PR	#4245
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux b106ac6f8705 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eccc29e`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/testReport/
Max. process+thread count	637 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4245/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

This reverts commit eccc29e.

* hdfs_find uses u_int32_t type for storing the value for the max-depth command line argument. * The type u_int32_t isn't standard, isn't available on Windows and thus breaks cross-platform compatibility. We need to replace this with uint32_t which is available on all platforms since it's part of the C++ standard.

GauthamBanasandra added the HDFS label Apr 28, 2022

GauthamBanasandra self-assigned this Apr 28, 2022

GauthamBanasandra requested a review from goiri April 28, 2022 17:08

goiri approved these changes Apr 28, 2022

View reviewed changes

...s-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs_smoke.cc Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

GauthamBanasandra force-pushed the fix-uint32 branch from 2eef94c to 63293bb Compare April 30, 2022 11:51

This comment was marked as outdated.

Sign in to view

GauthamBanasandra mentioned this pull request May 1, 2022

HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)" #4253

Closed

4 tasks

GauthamBanasandra added 2 commits May 3, 2022 09:17

Use uint32_t instead of u_int32_t

eb50be0

* Need to use uint32_t since it's available on all platform as it's part of the C++ standard.

Allow tests to run

eccc29e

GauthamBanasandra force-pushed the fix-uint32 branch from 63293bb to eccc29e Compare May 3, 2022 03:47

Revert "Allow tests to run"

a828e23

This reverts commit eccc29e.

GauthamBanasandra merged commit d346be9 into apache:trunk May 4, 2022

GauthamBanasandra deleted the fix-uint32 branch May 4, 2022 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16564. Use uint32_t for hdfs_find #4245

HDFS-16564. Use uint32_t for hdfs_find #4245

GauthamBanasandra commented Apr 28, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Apr 30, 2022

This comment was marked as outdated.

GauthamBanasandra commented Apr 30, 2022

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022

HDFS-16564. Use uint32_t for hdfs_find #4245

HDFS-16564. Use uint32_t for hdfs_find #4245

Conversation

GauthamBanasandra commented Apr 28, 2022

Description of PR

How was this patch tested?

For code changes:

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Apr 30, 2022

This comment was marked as outdated.

GauthamBanasandra commented Apr 30, 2022

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022

hadoop-yetus commented May 3, 2022