HADOOP-18180. Replace use of twitter util-core with java futures #4115

pjfanning · 2022-03-28T13:15:56Z

Description of PR

HADOOP-18180 replace use of twitter util-core - it uses scala 2.11 and will cause issue for Spark and other scala based tools that use different versions of scala

How was this patch tested?

CI build
local build

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

pjfanning · 2022-03-28T13:21:01Z

@steveloughran would you be able to review this when you have time?

steveloughran · 2022-03-28T14:35:35Z

did you run the integration tests? we have to be ruthless here about the test process for anything which goes near the object stores, as yetus doesn't have any cloud credentials

steveloughran

looks good, i'm pleased to see how easy it is to get off the twitter libs, as having scala on the cp was going to be a blocker for the merge.

please tell us the s3 endpoint you ran the integration tests against
once we are happy with the fs.common package, i plan to move it to hadoop-common and into o.a.h.fs.impl. leaving it in hadoop-aws for now makes dev a bit more agile (no need to compile two jars for each change). something to bear in mind, as this stuff is more broadly useful

steveloughran · 2022-03-28T14:37:04Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

@@ -433,13 +434,13 @@ private void addToCacheAndRelease(BufferData data, Future<Void> blockFuture) {
    }

    try {
-      Await.result(blockFuture);
+      blockFuture.get(); //TODO consider calling get(long timeout, TimeUnit unit) instead


if there's a todo item here, it'll need fixing or at least understanding of why a fix isn't needed

would it be possible to agree a timeout value - maybe 1 hour? - otherwise, this could block indefinitely

this is also the same problem that would occur with the previous Await.result call

I updated this with a 1 hour timeout

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/ExecutorServiceFuturePool.java

pjfanning · 2022-03-28T19:37:27Z

did you run the integration tests? we have to be ruthless here about the test process for anything which goes near the object stores, as yetus doesn't have any cloud credentials

I must admit to only having tested this with unit tests. I'm not really an expert on running hadoop integration tests. Would https://github.com/findify/s3mock or something similar be an option?

hadoop-yetus · 2022-03-29T00:24:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	33m 31s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 47s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 39s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 46s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 27s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 36s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 10s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	20m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 8s		the patch passed
+1 💚	shadedclient	20m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		88m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/8/artifact/out/Dockerfile
GITHUB PR	#4115
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux 68c1ec9a8203 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `915b1d3`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/8/testReport/
Max. process+thread count	629 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

minor comments, we are nearly done. for yetus;s complaints, a @SuppressWarnings("unchecked") should do it

steveloughran · 2022-03-29T12:32:16Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

@@ -354,7 +355,7 @@ private void readBlock(BufferData data, boolean isPrefetch, BufferData.State...
    }

    @Override
-    public Void applyE() {
+    public Void get() {


we need to think what we want to do here.

we would want errors to be counted and included in stream stats

but we've seen abfs apps flooded with stack traces when a transient network error breaks every prefetch

its good to be aware of transient errors, but not worry too much until the final reaqd.

created https://issues.apache.org/jira/browse/HADOOP-18185

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingBlockManager.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

...ools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java

steveloughran · 2022-03-29T12:45:08Z

...ools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java

+      Future<Void> future = futurePool.executeFunction(() -> {
+        throw new IllegalStateException("deliberate");
+      });
+      assertThrows(ExecutionException.class, () -> future.get(30, TimeUnit.SECONDS));


prefer LambdaTestUtils intercept, which includes the toString value of the operation on a failure.

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/read/TestS3InputStream.java

steveloughran · 2022-03-29T12:50:43Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/ExecutorServiceFuturePool.java

+public class ExecutorServiceFuturePool {
+  private ExecutorService executor;
+
+  public ExecutorServiceFuturePool(ExecutorService executor) {


interesting q here about what service to support/integrate with here, especially for faireness in processs with many readers.

created https://issues.apache.org/jira/browse/HADOOP-18186 as a followup to this.

steveloughran · 2022-03-29T12:58:56Z

hi, the testing process is in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md ; takes about 15 min from your laptop; cleans up the bucket afterwards so there's no long term billing.

you can try against a local mino server; do note some outstanding quirks there (https://issues.apache.org/jira/browse/HADOOP-18019 ).

hadoop-yetus · 2022-03-29T22:44:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	35m 10s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 47s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 38s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 44s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 26s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 33s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 8s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	20m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 12s		the patch passed
+1 💚	shadedclient	21m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		91m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/10/artifact/out/Dockerfile
GITHUB PR	#4115
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux 64d95fb9a343 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `57a5ccb`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/10/testReport/
Max. process+thread count	546 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-30T20:00:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	33m 59s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 47s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 40s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 46s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 28s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 9s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	20m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 35s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 10s		the patch passed
+1 💚	shadedclient	21m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 27s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		90m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/12/artifact/out/Dockerfile
GITHUB PR	#4115
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux 72f31be0d4ea 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `ca715d4`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/12/testReport/
Max. process+thread count	629 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

pjfanning · 2022-03-30T20:12:33Z

@steveloughran @monthonk thanks for checking the PR. I'm not really a hadoop-aws user. I just saw the coding problem and being familiar with Scala and Java asynch programming, I thought I'd try my hand at a patch.
I tried running the integration tests but they just hang. It seems like the config I set up is wrong but I don't really have the time to debug this. If someone else wants to use this PR as a template feel free but I think I will abandon my efforts - as I have a lot of other stuff on at the moment.

hadoop-yetus · 2022-03-30T21:01:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	34m 26s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 46s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 39s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 45s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 27s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 8s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	20m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	20m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 28s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		89m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/13/artifact/out/Dockerfile
GITHUB PR	#4115
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux be2d87d600ae 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `c01c7e0`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/13/testReport/
Max. process+thread count	692 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/13/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

pjfanning · 2022-04-01T19:26:51Z

reopened due a request from @steveloughran

hadoop-yetus · 2022-04-01T20:56:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	33m 35s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 46s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 40s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 45s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 28s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 9s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	20m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	20m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 30s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		88m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/14/artifact/out/Dockerfile
GITHUB PR	#4115
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux 2e8e7c7b617e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `c01c7e0`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/14/testReport/
Max. process+thread count	545 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4115/14/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

dannycjones · 2022-04-04T13:54:38Z

Looking at the test results shared by Monthon...

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestS3ARequesterPays.testRequesterPaysOptionSuccess:78 [Counter named stream_read_opened] 
Expecting:
 <0L>
to be greater than:
 <1L> 
[ERROR]   ITestS3AUnbuffer.testUnbuffer:75->Assert.assertTrue:53->Assert.assertTrue:42->Assert.fail:87
[ERROR]   ITestS3AUnbuffer.testUnbufferStreamStatistics:149->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Count of stream_read_bytes starting=0 current=0 diff=0: stream_read_bytes expected:<8> but was:<0>
[ERROR]   ITestMarkerTool.testRunLimitedLandsatAudit:320->AbstractMarkerToolTest.runToFailure:271 Expected an exception of type class org.apache.hadoop.util.ExitUtil$ExitException
[ERROR] Errors: 
[ERROR]   ITestS3AContractUnbuffer>AbstractContractUnbufferTest.testUnbufferOnClosedFile:90->AbstractContractUnbufferTest.unbuffer:124 » IO
[ERROR]   ITestS3AContractStreamIOStatistics>AbstractContractStreamIOStatisticsTest.testInputStreamStatisticRead:253 » EOF
[INFO] 
[ERROR] Tests run: 1069, Failures: 4, Errors: 2, Skipped: 185

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestS3AFileContextStatistics>FCStatisticsBaseTest.testStatistics:107 » EOF At...
[INFO] 
[ERROR] Tests run: 108, Failures: 0, Errors: 1, Skipped: 68

I think all but one of these are expected - ITestMarkerTool.testRunLimitedLandsatAudit:320 is a known issue in trunk (HADOOP-18168) and 4 of those are known to fail in this feature branch (HADOOP-18175).

The interesting one is not mentioned in HADOOP-18175 is ITestS3AContractStreamIOStatistics>AbstractContractStreamIOStatisticsTest.testInputStreamStatisticRead:253 » EOF.

steveloughran · 2022-04-04T14:16:40Z

unbuffer happens on some networks, marker is known. let me try locally.

this is a feature branch and I'm not too worried about patches introducing failures, especially this one, which must go in

steveloughran · 2022-04-04T14:42:07Z

stack of the failure

[ERROR] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)  Time elapsed: 1.591 s  <<< ERROR!
java.io.EOFException: Attempted to seek or read past the end of the file 512
	at org.apache.hadoop.fs.s3a.read.S3InputStream.throwIfInvalidSeek(S3InputStream.java:440)
	at org.apache.hadoop.fs.s3a.read.S3InputStream.seek(S3InputStream.java:235)
	at org.apache.hadoop.fs.s3a.read.S3PrefetchingInputStream.seek(S3PrefetchingInputStream.java:162)
	at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:85)
	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:98)
	at org.apache.hadoop.fs.FCStatisticsBaseTest.testStatistics(FCStatisticsBaseTest.java:107)

it's happening in the seek(oldPos) code in the default implementation of positioned read, because the previous unpositioned read read the whole file, so pos() is at file length.

    byte[] buf = new byte[blockSize];
    int bytesRead = fstr.read(buf, 0, blockSize);
    fstr.read(0, buf, 0, blockSize);  *here*

This is quite a good little failure; I'm impressed but who ever managed to think at this corner case. pos is at EOF, so returning to it is raising the EOF.

It is unrelated to this PR; I will file a JIRA. We will probably want seek() to not do the checks, and have read() do the validation, which is mostly what s3a does , as it only rejects negative offsets.

steveloughran · 2022-04-04T14:45:54Z

+1 merged.

only thought after i'd hit the merge that i should include a reference to the HADOOP-18028 jira in the message; I will do that in the others, so a log grep will find them all.

…3A prefetching stream (apache#4115) Contributed by PJ Fanning.

This is the the a rollup patch of the HADOOP-18028 S3A performance input stream feature branch. Contains HADOOP-18028. High performance S3A input stream (apache#4109) This is the the merge of the HADOOP-18028 S3A performance input stream. This patch on its own is incomplete and must be accompanied by all other commits with HADOOP-18028 in their git commit message. Consult the JIRA for that list Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures in S3A prefetching stream (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail. Change-Id: I48f217086531c12d6e2f0f91e39f17054a74d20f

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I6511c51c3580c57eb72e8ea686c88e3917d12a06

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I3eca19564dc0c0cb83184f4a42605dbafd908937

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

HADOOP-18080 replace use of twitter util-core

d48eaa7

pjfanning mentioned this pull request Mar 28, 2022

[HADOOP-18180] remove use of twitter util-core (scala jar) #4114

Closed

4 tasks

pjfanning changed the title ~~HADOOP-18080 replace use of twitter util-core~~ HADOOP-18180 replace use of twitter util-core Mar 28, 2022

steveloughran self-assigned this Mar 28, 2022

steveloughran reviewed Mar 28, 2022

View reviewed changes

steveloughran changed the title ~~HADOOP-18180 replace use of twitter util-core~~ HADOOP-18180. Replace use of twitter util-core with java futures Mar 28, 2022

pjfanning added 2 commits March 28, 2022 16:54

address some review comments

02be95b

add tests

a0dd7fe

pjfanning added 5 commits March 28, 2022 21:38

missing license

bcab50d

rename FuturePool methods

339f47d

Update ExecutorServiceFuturePool.java

c5b3c4b

Update ExecutorServiceFuturePool.java

5f9b643

fix indents

915b1d3

apache deleted a comment from hadoop-yetus Mar 29, 2022

steveloughran requested changes Mar 29, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Mar 29, 2022

pjfanning added 3 commits March 29, 2022 22:43

PR review items

27138bd

use LambdaTestUtils

57a5ccb

checkstyle warnings

66562cc

pjfanning added 2 commits March 30, 2022 20:28

Update ExecutorServiceFuturePool.java

ca715d4

try to match import order

c01c7e0

pjfanning marked this pull request as draft March 30, 2022 20:08

pjfanning closed this Mar 30, 2022

pjfanning reopened this Apr 1, 2022

pjfanning marked this pull request as ready for review April 4, 2022 14:24

steveloughran merged commit 3aa03e0 into apache:feature-HADOOP-18028-s3a-prefetch Apr 4, 2022

pjfanning deleted the HADOOP-18180-remove-twitter-util branch April 4, 2022 14:48

ahmarsuhail pushed a commit to ahmarsuhail/hadoop that referenced this pull request May 19, 2022

HADOOP-18180. Replace use of twitter util-core with java futures in S…

9ebfcce

…3A prefetching stream (apache#4115) Contributed by PJ Fanning.

steveloughran pushed a commit to steveloughran/hadoop that referenced this pull request Jul 28, 2022

HADOOP-18180. Replace use of twitter util-core with java futures in S…

5a1f4dd

…3A prefetching stream (apache#4115) Contributed by PJ Fanning.

steveloughran mentioned this pull request Aug 17, 2022

HADOOP-18028. High performance S3A input stream #4752

Merged

4 tasks

steveloughran mentioned this pull request Apr 14, 2023

HADOOP-18028. High performance S3A input stream (#4752) #5559

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18180. Replace use of twitter util-core with java futures #4115

HADOOP-18180. Replace use of twitter util-core with java futures #4115

pjfanning commented Mar 28, 2022 •

edited

Loading

pjfanning commented Mar 28, 2022

steveloughran commented Mar 28, 2022

steveloughran left a comment

steveloughran Mar 28, 2022

pjfanning Mar 28, 2022

pjfanning Mar 28, 2022

pjfanning Mar 28, 2022

pjfanning commented Mar 28, 2022

hadoop-yetus commented Mar 29, 2022

steveloughran left a comment

steveloughran Mar 29, 2022

steveloughran Mar 29, 2022

steveloughran Mar 29, 2022

steveloughran commented Mar 29, 2022

hadoop-yetus commented Mar 29, 2022

hadoop-yetus commented Mar 30, 2022

pjfanning commented Mar 30, 2022

hadoop-yetus commented Mar 30, 2022

pjfanning commented Apr 1, 2022

hadoop-yetus commented Apr 1, 2022

dannycjones commented Apr 4, 2022

steveloughran commented Apr 4, 2022

steveloughran commented Apr 4, 2022

steveloughran commented Apr 4, 2022

HADOOP-18180. Replace use of twitter util-core with java futures #4115

HADOOP-18180. Replace use of twitter util-core with java futures #4115

Conversation

pjfanning commented Mar 28, 2022 • edited Loading

Description of PR

How was this patch tested?

For code changes:

pjfanning commented Mar 28, 2022

steveloughran commented Mar 28, 2022

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Mar 28, 2022

Choose a reason for hiding this comment

pjfanning Mar 28, 2022

Choose a reason for hiding this comment

pjfanning Mar 28, 2022

Choose a reason for hiding this comment

pjfanning Mar 28, 2022

Choose a reason for hiding this comment

pjfanning commented Mar 28, 2022

hadoop-yetus commented Mar 29, 2022

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Mar 29, 2022

Choose a reason for hiding this comment

steveloughran Mar 29, 2022

Choose a reason for hiding this comment

steveloughran Mar 29, 2022

Choose a reason for hiding this comment

steveloughran commented Mar 29, 2022

hadoop-yetus commented Mar 29, 2022

hadoop-yetus commented Mar 30, 2022

pjfanning commented Mar 30, 2022

hadoop-yetus commented Mar 30, 2022

pjfanning commented Apr 1, 2022

hadoop-yetus commented Apr 1, 2022

dannycjones commented Apr 4, 2022

steveloughran commented Apr 4, 2022

steveloughran commented Apr 4, 2022

steveloughran commented Apr 4, 2022

pjfanning commented Mar 28, 2022 •

edited

Loading