HADOOP-18028. High performance S3A input stream

[steveloughran]

Description of PR

This is the PR of #3736 applied to a dedicated feature branch, with some minor pre-merge tuning with all subsequent changes to be their own PR

• rename test classes, have AbstractHadoopTestBase as the base
• package info files for new packages
• import ordering
• move to intercept() for assertions; ExceptionAsserts is invoking it and can be removed in future.

this adds a dependency on a twitter lib which looks like scala code. that MUST be cut before we can merge.

How was this patch tested?

s3 london with -Dparallel-tests -DtestsThreadCount=8 -Dmarkers=keep -Dscale

there are some failing integration tests which will need to be fixed before the feature branch is merged to trunk.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	34m 21s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 43s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 37s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 23s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 41s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 24s		feature-HADOOP-18028-s3a-prefetch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 30s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 7s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	21m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 40s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		hadoop-tools/hadoop-aws: The patch generated 0 new + 5 unchanged - 2 fixed = 5 total (was 7)
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 9s		the patch passed
+1 💚	shadedclient	20m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 30s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		89m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4109/1/artifact/out/Dockerfile
GITHUB PR	#4109
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle markdownlint
uname	Linux 188f1fde1f74 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / c888527
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4109/1/testReport/
Max. process+thread count	549 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4109/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[pjfanning]

@steveloughran twitter futures should be fairly easily replaced with Java futures

• this seems to a fairly good description of the twitter FuturePool - https://blog.knoldus.com/twitter-future-in-scala/
• should be possible to port it to Java easily enough
• it may even not be necessary to port the twitter FuturePool because it appears its main use to help Scala users avoid having to deal Java futures used in AWS code (cf https://medium.com/@sveinnfannar/thread-pooling-in-scala-using-twitter-s-futurepool-8f38c6448fa2)

I have an example change - steveloughran#2