HADOOP-18231. Fixes failing tests & drain stream async. #4386

ahmarsuhail · 2022-05-31T12:43:53Z

Description of PR

This is a combined PR for:

Add new test for Prefetching:

JIRA. Previous PR before rebase was opened here.

ITestS3AInputStreamPerformance was failing when prefetching is enabled. This PR disables prefetching when running ITestS3AInputStreamPerformance and adds in a new test class ITestS3PrefetchingInputStream with tests specific for prefetching.

Drain stream async:

JIRA. Previous PR before rebase was opened here.

If we close the prefetching input stream before prefetched blocks have finished reading the S3 input stream, the sdk repeatedly complains "Not all bytes were read from the S3ObjectInputStream". This happened on S3AInputStream as well, see this issue for more details. Closing the stream before draining it will abort the connection, so to allow for connection reuse we drain it asynchronously when remaining bytes is > async drain threshold.

Fix ITestS3AOpenCost:

JIRA

This test was failing after the rebase.

testOpenFileShorterLength fails because inputStreamStats need to be created before opening the stream so that the time to prepare to open the file is included.

testOpenFileLongerLength fails because in this read method when for eg: file length is 8 and length to read is 18, it swallows the EOF exception and returns the num of successful bytes read which will be 8. So readFully doesn't throw an EOF immediately but then tries to read bytes 8-18 again, and then fails.

How was this patch tested?

Tested in eu-west-1 by running

mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

steveloughran · 2022-06-09T11:57:10Z

made some comments. looked at the EOF swallowing code...i think we need to be happy there that all is good in terms of handling recoverable failures but not retrying on nonrecoverable ones.

steveloughran

made some minor suggestions, created a jira on "need to make sure retry code is good", which is very much manual review work, relying on @retries annotations for guidance

steveloughran · 2022-06-09T08:47:56Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

@@ -64,6 +67,12 @@ public class S3File implements Closeable {
  // That allows us to close the object when closing the stream.
  private Map<InputStream, S3Object> s3Objects;

+  // uri of the object being read


nit: can you make these all javadocs, so they show up in IDEs.

steveloughran · 2022-06-09T08:48:35Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

@@ -98,6 +107,7 @@ public S3File(
    this.streamStatistics = streamStatistics;
    this.changeTracker = changeTracker;
    this.s3Objects = new IdentityHashMap<InputStream, S3Object>();


while you are there, can you change to a simple <> here

steveloughran · 2022-06-09T08:51:59Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

    Io.closeIgnoringIoException(inputStream);
-    Io.closeIgnoringIoException(obj);
+    Io.closeIgnoringIoException(requestObject);


I'm not sure which library this is. switch to our IOUtils.cleanupWithLogger(lOG, inputStream, requestObject)

this will log any exceptions at debug, if ever needed

This was a new Io class added as part of the initial prefetching PR. I've removed this class, and updated usages with cleanupWithLogger

steveloughran · 2022-06-09T08:54:49Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3InputStream.java


    Validate.checkNotNull(context, "context");
    Validate.checkNotNull(s3Attributes, "s3Attributes");
    Validate.checkNotNull(client, "client");
+    Validate.checkNotNull(streamStatistics, "streamStatistics");


this is one of those things we'd inevitably want to clean up. i can see a "steve does a cleanup" PR coming soon....
If you use Objects.requireNonNull() you can add the check on L119 so no need for extra work...its where new code should be going, even if older code still uses the guava checkNotNull

steveloughran · 2022-06-09T08:57:57Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3PrefetchingInputStream.java

+  private static final int _1K = 1024;
+  // Path for file which should have length > block size so S3CachingInputStream is used
+  private Path largeFile;
+  private FileSystem fs;


can you name this largeFileFS to make clear why it is being used

steveloughran · 2022-06-09T11:28:48Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3PrefetchingInputStream.java

+ * Test the prefetching input stream, validates that the underlying S3CachingInputStream and
+ * S3InMemoryInputStream are working as expected.
+ */
+public class ITestS3PrefetchingInputStream extends AbstractS3ACostTest {


can you call this ITestS3PrefetchingInputStream? I want to rename the other classes too, but let's start with the new ones

guessing you meant something else as it's already called ITestS3PrefetchingInputStream?

steveloughran · 2022-06-09T11:30:32Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3PrefetchingInputStream.java

+    super(true);
+  }
+
+  private static final int _1K = 1024;


I think hadoop common test should have a version of
org.apache.hadoop.fs.azure.integration.Sizes which we can reference here and elsewhere. it uses the S_ prefix to keep checkstyle quiet.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

steveloughran · 2022-06-09T11:32:23Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

+   * @param inputStream   stream to close.
+   * @return was the stream aborted?
+   */
+  private boolean drainOrAbortHttpStream(boolean shouldAbort, final String reason,


style nit: can you split one to a line, all tagged final. there's enough params to justify the effort

steveloughran · 2022-06-09T11:38:47Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3PrefetchingInputStream.java

+    try (FSDataInputStream in = fs.open(largeFile)) {
+      IOStatistics ioStats = in.getIOStatistics();
+
+      byte[] buffer = new byte[(int) largeFileSize];


this is trouble if that test file is really big. prefer to create a smaller buffer and iterate through with readFully calls.

now, one thing we don't check reliably, even in AbstractSTestS3AHugeFiles, is are the contents of large files constant even if you read them in different ways?

i think we need to worry about this, though also think AbstractSTestS3AHugeFiles might be the place...we can build an md5 checksum on the upload and verify on the reads.

ahmarsuhail · 2022-06-09T15:48:28Z

@steveloughran thanks for reviewing, I've updated as per your comments

hadoop-yetus · 2022-06-09T17:30:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	42m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	1m 0s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 51s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 44s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	1m 0s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 42s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 43s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 39s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	24m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 42s		the patch passed
+1 💚	compile	0m 49s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 49s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 26s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 3 new + 14 unchanged - 1 fixed = 17 total (was 15)
+1 💚	mvnsite	0m 44s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 26s		the patch passed
+1 💚	shadedclient	23m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 5s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		125m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/6/artifact/out/Dockerfile
GITHUB PR	#4386
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 452c5c56edcf 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `ebb1b1d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/6/testReport/
Max. process+thread count	523 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-06-10T15:56:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	41m 33s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	1m 1s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 44s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 58s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 50s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 50s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 43s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	28m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 45s		the patch passed
+1 💚	compile	0m 50s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 50s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 28s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 27s		the patch passed
+1 💚	shadedclient	26m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 2s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		114m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/10/artifact/out/Dockerfile
GITHUB PR	#4386
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 01a1f9042d56 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `eca1cba`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/10/testReport/
Max. process+thread count	622 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-06-13T10:34:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	40m 18s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 55s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 43s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 54s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 41s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 42s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 30s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	24m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 43s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 23s		hadoop-tools/hadoop-aws: The patch generated 0 new + 14 unchanged - 1 fixed = 14 total (was 15)
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 14s		the patch passed
+1 💚	shadedclient	23m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 48s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		104m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/11/artifact/out/Dockerfile
GITHUB PR	#4386
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux cd39d9599199 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `31febbf`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/11/testReport/
Max. process+thread count	594 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1. pending one minor change

the new test still needs a trailing newline to be added after the final "}" ... I think you added it above (which is also good)

add that and it's ready to merge. I actually want to test this out this week with the cloudstore's bandwidth command
https://github.com/steveloughran/cloudstore#command-bandwidth

steveloughran · 2022-06-14T17:54:47Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3File.java

@@ -284,8 +308,8 @@ private boolean drainOrAbortHttpStream(boolean shouldAbort, final String reason,
        shouldAbort = true;
      }
    }
-    Io.closeIgnoringIoException(inputStream);
-    Io.closeIgnoringIoException(requestObject);
+    cleanupWithLogger(LOG, inputStream);


FYI, you can pass any number of Closeables in; not worth updating the PR for, but useful to know

steveloughran · 2022-06-14T17:56:29Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3PrefetchingInputStream.java

+    }
+  }
+
+}


still needs a trailing newline after the closing bracket;

Should I look into adding this as a CheckStyle rule? I can open a JIRA if so.

Looks like there's one to fit this: https://checkstyle.sourceforge.io/config_misc.html#NewlineAtEndOfFile

ahmarsuhail · 2022-06-15T10:45:17Z

thanks @steveloughran, have added in the new line

hadoop-yetus · 2022-06-15T12:28:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	40m 26s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 58s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 43s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 54s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 40s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 43s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	23m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 43s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 24s		hadoop-tools/hadoop-aws: The patch generated 0 new + 14 unchanged - 1 fixed = 14 total (was 15)
+1 💚	mvnsite	0m 41s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 14s		the patch passed
+1 💚	shadedclient	23m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 49s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		103m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/12/artifact/out/Dockerfile
GITHUB PR	#4386
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux d1da46f94e71 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `448a81c`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/12/testReport/
Max. process+thread count	524 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4386/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

dannycjones · 2022-07-01T11:26:41Z

Hey @steveloughran, we have a few prefetching PRs starting to get blocked on this one which fixes some unit tests. Can you take a look next week?

steveloughran

+1, merging. thanks!

apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail

This is the the a rollup patch of the HADOOP-18028 S3A performance input stream feature branch. Contains HADOOP-18028. High performance S3A input stream (apache#4109) This is the the merge of the HADOOP-18028 S3A performance input stream. This patch on its own is incomplete and must be accompanied by all other commits with HADOOP-18028 in their git commit message. Consult the JIRA for that list Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures in S3A prefetching stream (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail. Change-Id: I48f217086531c12d6e2f0f91e39f17054a74d20f

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I6511c51c3580c57eb72e8ea686c88e3917d12a06

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I3eca19564dc0c0cb83184f4a42605dbafd908937

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

ahmarsuhail added 6 commits May 31, 2022 13:38

adds in new test for prefetching input stream

3e5113f

creates streamStats before opening stream

13d181f

updates numBlocks calculation method

74813fe

fixes ITestS3AOpenCost.testOpenFileLongerLength

d96babe

drains stream async

fbe57d4

fixes failing unit test

52e6f96

ahmarsuhail changed the title ~~adds in new test for prefetching input stream~~ HADOOP-18231. Fixes failing tests & drain stream async. Jun 8, 2022

steveloughran requested changes Jun 9, 2022

View reviewed changes

ahmarsuhail added 2 commits June 9, 2022 16:23

updates as per review comments

0807cbd

use requireNonNull

ebb1b1d

removes extra new line

a5f32e7

ahmarsuhail added 2 commits June 10, 2022 07:25

fixes checkstyle errors

115c4e0

fix typo

eca1cba

removes unused import

31febbf

apache deleted a comment from hadoop-yetus Jun 14, 2022

steveloughran approved these changes Jun 14, 2022

View reviewed changes

adds in new line

448a81c

This was referenced Jun 18, 2022

HADOOP-18190. Adds iostats for prefetching #4458

Merged

HADOOP-18254. Disable prefetching by default. #4469

Merged

ahmarsuhail mentioned this pull request Jun 30, 2022

HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

Closed

steveloughran approved these changes Jul 15, 2022

View reviewed changes

steveloughran merged commit 18d4d32 into apache:feature-HADOOP-18028-s3a-prefetch Jul 15, 2022

steveloughran mentioned this pull request Aug 17, 2022

HADOOP-18028. High performance S3A input stream #4752

Merged

4 tasks

steveloughran mentioned this pull request Apr 14, 2023

HADOOP-18028. High performance S3A input stream (#4752) #5559

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18231. Fixes failing tests & drain stream async. #4386

HADOOP-18231. Fixes failing tests & drain stream async. #4386

ahmarsuhail commented May 31, 2022 •

edited

Loading

steveloughran commented Jun 9, 2022

steveloughran left a comment

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

ahmarsuhail Jun 9, 2022

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

ahmarsuhail Jun 9, 2022

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

steveloughran Jun 9, 2022

ahmarsuhail commented Jun 9, 2022

hadoop-yetus commented Jun 9, 2022

hadoop-yetus commented Jun 10, 2022

hadoop-yetus commented Jun 13, 2022

steveloughran left a comment

steveloughran Jun 14, 2022

steveloughran Jun 14, 2022

dannycjones Jul 1, 2022

ahmarsuhail commented Jun 15, 2022

hadoop-yetus commented Jun 15, 2022

dannycjones commented Jul 1, 2022

steveloughran left a comment

+                  }
+                }
+              }

HADOOP-18231. Fixes failing tests & drain stream async. #4386

HADOOP-18231. Fixes failing tests & drain stream async. #4386

Conversation

ahmarsuhail commented May 31, 2022 • edited Loading

Description of PR

Add new test for Prefetching:

Drain stream async:

Fix ITestS3AOpenCost:

How was this patch tested?

steveloughran commented Jun 9, 2022

steveloughran left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmarsuhail commented Jun 9, 2022

hadoop-yetus commented Jun 9, 2022

hadoop-yetus commented Jun 10, 2022

hadoop-yetus commented Jun 13, 2022

steveloughran left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmarsuhail commented Jun 15, 2022

hadoop-yetus commented Jun 15, 2022

dannycjones commented Jul 1, 2022

steveloughran left a comment

Choose a reason for hiding this comment

ahmarsuhail commented May 31, 2022 •

edited

Loading