HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 #1160

steveloughran · 2019-07-25T13:21:36Z

Improving diagnostics of scanning

stack traces don't get lost as failures get wrapped/aggregated
debug level logging of what is up in Globber
Preparation for a test of LocatedFileStatus in S3A, though I've got some better ideas there (i.e. make it a scale test)
Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

steveloughran · 2019-07-29T18:29:03Z

tested: s3 ireland with -Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Ddynamodb -Dnonauth -Dscale

Also, check out cloudstore Release: 2019-07-29, whose locatefiles API call invokes this codepath and lets you see what the debugging is up to, including printing out the statistics of operations called.

steveloughran · 2019-07-30T16:29:31Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestLocatedFileStatusFetcher.java

+ * There's basic tests in ITestS3AFSMainOperations; this
+ * is see if we can create better corner cases.
+ */
+public class ITestLocatedFileStatusFetcher extends AbstractS3ATestBase {


Either I implement a test here or I cut the file

If you already started to implement the test then it would be useful.

I cut the file; all tests live in the ITestRestrictedRead operation.

Reason: a basic scan isn't that interesting, trying to break it is, and the test bucket I was working with was giving me list access to the store but not read on files. I wanted to see if that was the problem. i..e. it was the reporting that was at fault. It isn't, but there's a good sequence of work there combining basic LocatedStatusFetcher operations which work with those that don't

bgaborg · 2019-07-31T13:18:00Z

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestLocatedFileStatusFetcher.java

+ * There's basic tests in ITestS3AFSMainOperations; this
+ * is see if we can create better corner cases.
+ */
+public class ITestLocatedFileStatusFetcher extends AbstractS3ATestBase {


If you already started to implement the test then it would be useful.

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

bgaborg · 2019-07-31T13:43:29Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

+    intercept(AccessDeniedException.class, () ->
+        ContractTestUtils.readUTF8(roleFS, subdirFile, data.length));
+    if (authMode) {
+      // auth mode doesn't just not check the store,


doesn't just not -- maybe not the best wording

bgaborg · 2019-07-31T14:02:36Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

+          ContractTestUtils.readUTF8(roleFS, emptyFile, 0));
+    }
+
+    // Fun with Glob


I would move this to another test. If something fails somebody (me) will need to debug and it could be harder for a test this long.

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java

bgaborg · 2019-07-31T14:07:15Z

I've also tested this against ireland and got a new test error:

[ERROR] Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 60.539 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 30.759 s  <<< FAILURE!
java.lang.AssertionError:
Expected no results from listLocatedStatus(/), but got 1 elements:
S3ALocatedFileStatus{path=s3a://gabota-versioned-bucket-ireland/fork-0003; isDirectory=true; modification_time=1564580542740; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.assertNoElements(AbstractContractRootDirectoryTest.java:218)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:202)
	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 1.527 s  <<< FAILURE!
java.lang.AssertionError:
listStatus(/) vs listLocatedStatus(/) with
listStatus =S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null listLocatedStatus = S3ALocatedFileStatus{path=s3a://gabota-versioned-bucket-ireland/fork-0003; isDirectory=true; modification_time=1564580544503; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}
S3ALocatedFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=1564580544503; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} expected:<1> but was:<2>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Note that this only fails after the parallel test run. This maybe related to this change, as I have not seen this fail yet. I will try to debug this using my WIP fsck.

I created https://issues.apache.org/jira/browse/HADOOP-16476 for a failure I've seen already to fail intermittently, but that should be unrelated.

steveloughran · 2019-07-31T14:09:08Z

Updated PR
*remove unused test suite

rebased to trunk
split the too-long test case in IITestRestrictedReadAccess into separate pieces, using fields to propagate paths between them. It will still fail fast on the first assert failure, but as the operations and assertions move up the set of methods used in a LocatedFileStatusFetcher scan, that's OK. If an early assert fails, so will the final one
added some specific intercept wrappers to improve legibility.
javadocs

No changes to production code.

Tested, S3 Ireland. All well except for HADOOP-16477, which surface as I'd set the fs.s3a.encryption.key for the bucket. Unrelated

steveloughran · 2019-07-31T18:09:06Z

Unless this test is confusing DDB so there's no cleanup, I'd point to those test failures less as a sign of a problem here and more that our test runs still seem somehow to cause confusion, even after the tombstone fixes.

ITestS3GuardDDBRootOperations will save the results of its scans to somewhere under target/ which you can use to see where a discrepancy has crept in. Usually its something in s3 which isn't in s3guard: an OOB event of some kind

bgaborg

Hey @steveloughran, there are some requested changes pending from me at ITestRestrictedReadAccess.java. Otherwise, it's a +1 from me.

steveloughran · 2019-08-28T14:54:25Z

thanks. Think I've addressed your comments; pushing up for yetus then I'll commit. Tested: S3 ireland

steveloughran · 2019-09-03T15:36:25Z

checkstyle is about an 81 char line where splitting it would make things less readable

./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java:255:            new InterruptedIOException("Interrupted while getting file statuses"): Line is longer than 80 characters (found 81). [LineLength]

hadoop-yetus · 2019-09-04T17:17:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	82	Maven dependency ordering for branch
+1	mvninstall	1104	trunk passed
+1	compile	1119	trunk passed
+1	checkstyle	166	trunk passed
+1	mvnsite	200	trunk passed
+1	shadedclient	1185	branch has no errors when building and testing our client artifacts.
+1	javadoc	147	trunk passed
0	spotbugs	73	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	320	trunk passed
		_ Patch Compile Tests _
0	mvndep	29	Maven dependency ordering for patch
+1	mvninstall	124	the patch passed
+1	compile	1008	the patch passed
+1	javac	1008	the patch passed
-0	checkstyle	148	root: The patch generated 1 new + 231 unchanged - 9 fixed = 232 total (was 240)
+1	mvnsite	179	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	693	patch has no errors when building and testing our client artifacts.
+1	javadoc	139	the patch passed
+1	findbugs	314	the patch passed
		_ Other Tests _
+1	unit	555	hadoop-common in the patch passed.
+1	unit	339	hadoop-mapreduce-client-core in the patch passed.
+1	unit	92	hadoop-aws in the patch passed.
+1	asflicense	44	The patch does not generate ASF License warnings.
		7972

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/15/artifact/out/Dockerfile
GITHUB PR	#1160
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0bb11b0954b6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1ae7759`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/15/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/15/testReport/
Max. process+thread count	1480 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/15/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java

hadoop-yetus · 2019-09-12T17:22:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	40	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	25	Maven dependency ordering for branch
+1	mvninstall	1019	trunk passed
+1	compile	981	trunk passed
+1	checkstyle	147	trunk passed
+1	mvnsite	190	trunk passed
+1	shadedclient	1074	branch has no errors when building and testing our client artifacts.
+1	javadoc	150	trunk passed
0	spotbugs	71	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	267	trunk passed
		_ Patch Compile Tests _
0	mvndep	27	Maven dependency ordering for patch
+1	mvninstall	114	the patch passed
+1	compile	926	the patch passed
+1	javac	926	the patch passed
-0	checkstyle	147	root: The patch generated 1 new + 229 unchanged - 9 fixed = 230 total (was 238)
+1	mvnsite	184	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	685	patch has no errors when building and testing our client artifacts.
+1	javadoc	151	the patch passed
+1	findbugs	290	the patch passed
		_ Other Tests _
+1	unit	541	hadoop-common in the patch passed.
+1	unit	335	hadoop-mapreduce-client-core in the patch passed.
+1	unit	82	hadoop-aws in the patch passed.
+1	asflicense	55	The patch does not generate ASF License warnings.
		7420

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/17/artifact/out/Dockerfile
GITHUB PR	#1160
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 17bd2e49da74 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `2ff2a7f`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/17/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/17/testReport/
Max. process+thread count	1370 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/17/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-27T19:47:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1822	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	69	Maven dependency ordering for branch
+1	mvninstall	1063	trunk passed
+1	compile	1019	trunk passed
+1	checkstyle	164	trunk passed
+1	mvnsite	188	trunk passed
+1	shadedclient	1146	branch has no errors when building and testing our client artifacts.
+1	javadoc	169	trunk passed
0	spotbugs	71	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	270	trunk passed
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
+1	mvninstall	112	the patch passed
+1	compile	969	the patch passed
+1	javac	969	the patch passed
-0	checkstyle	165	root: The patch generated 1 new + 229 unchanged - 9 fixed = 230 total (was 238)
+1	mvnsite	186	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	733	patch has no errors when building and testing our client artifacts.
+1	javadoc	169	the patch passed
+1	findbugs	297	the patch passed
		_ Other Tests _
+1	unit	545	hadoop-common in the patch passed.
+1	unit	338	hadoop-mapreduce-client-core in the patch passed.
+1	unit	82	hadoop-aws in the patch passed.
+1	asflicense	55	The patch does not generate ASF License warnings.
		9568

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/18/artifact/out/Dockerfile
GITHUB PR	#1160
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 987cfbf9b713 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ce58c05`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/18/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/18/testReport/
Max. process+thread count	1597 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1160/18/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

This adds the ITestS3AFSMainOperations test so that we have some globber tests against the S3A FS. They didn't find problems there, only a bug in setWorkingDir(). Change-Id: Idbdc4b049fa70fa31aa701b2e1c570c5bd208521

Improving diagnostics of scanning * stack traces don't get lost as failures get wrapped/aggregated * debug level logging of what is up in Globber * Preparation for a test of LocatedFileStatus in S3A, though I've got some better ideas there (i.e. make it a scale test) Change-Id: Ie85eef17a52314d35c95defa4105d72b8c84dd20

this eliminates HEAD requests on scans where candidates are files Change-Id: I56971977e28a3adf09cae3df1c3d6e634cbc7295

… no-read paths key point: LIST always works, but HEAD and GET fail; in auth mode many such failures get skipped. Change-Id: I66fc17021d5f2b4015c9afbb446df6cb4df0b9ce

… Store Lots of tests (bad: all same test case; will fix), to invoke LocatedFileStatusFetcher against various S3 paths Key summary: you get the error seen in the HADOOP-16458 JIRA if * the path contains no pattern and does not exist * the path contains no pattern, exists but doesn't match the supplied pattern filter. Change-Id: Ib66f316a526982136e1345d10e9750aaec48cb95

LocatedFileStatusFetcher * Debug level logging of every callable invoked, thread count and other useful events. * Minor IDE-related cleanups as this is the first time this file has been looked at for a while. * fixup javadocs Globber * moved to builder pattern to the new (and still marked as private) API For creating a globber. If this really delivers speedups against stores, we'd benefit from it in our own stores and externally in GCS, so ultimately it may become public. For now, just design it ready for that. Invoker * log duration of operations @ debug Change-Id: I87ca7008ab787a848a35457c7a72d8f47e6d558f

… off Change-Id: Ibe7f502a92c67040dbdba464e13751fdbe0db4f3

… method into pieces Change-Id: Ic65c49bd512f4b8f33c211b9e4924b203165619f

Change-Id: I6937aa4c7de8adc135d9eb6bf99cca1b69773014

…ileStatusFetcher Change-Id: I592cff79fce8680daee55bae983465919734a8f1

Change-Id: I792c6187fdaaf6c981ac635ec9b527a925ad3d14

Change-Id: I3c1d64749107e6f69be76cdcb4eab724cdb1f9ba

-break up the ITestRestrictedReadAccess operations into more methods and change test* to check* as the prefix as the IDE was complaining about test- prefixed methods which weren't annotated. -javadoc and other reviews of the Globber changes Change-Id: Ib4f176926e7631d93829bab45e85857480521042

Change-Id: I80e5054ae2a5b7c7b616586f9882f07fcf0b8f92

This is the first time anyone has edited the file for a few years; most of the warnings are unrelated to the patch. Change-Id: I429bc5f3c28d33f0321890487faa7ac2fd7a5ddd

steveloughran · 2019-10-01T17:11:57Z

finally merged to trunk. Thx for review

steveloughran added bug fs/s3 changes related to hadoop-aws; submitter must declare test endpoint work in progress PRs still Work in Progress; reviews not expected but still welcome labels Jul 25, 2019

steveloughran force-pushed the s3/HADOOP-16458-globber branch from 1fea0d3 to 5b41f7c Compare July 26, 2019 23:49

steveloughran requested a review from bgaborg July 29, 2019 18:26

steveloughran commented Jul 30, 2019

View reviewed changes

bgaborg requested changes Jul 31, 2019

View reviewed changes

steveloughran force-pushed the s3/HADOOP-16458-globber branch from afe8030 to 9067458 Compare July 31, 2019 14:05

steveloughran removed the work in progress PRs still Work in Progress; reviews not expected but still welcome label Jul 31, 2019

bgaborg requested changes Aug 16, 2019

View reviewed changes

apache deleted a comment from hadoop-yetus Aug 28, 2019

steveloughran force-pushed the s3/HADOOP-16458-globber branch from 9067458 to 861a19d Compare August 28, 2019 16:32

apache deleted a comment from hadoop-yetus Sep 3, 2019

steveloughran force-pushed the s3/HADOOP-16458-globber branch from 861a19d to 0ece51b Compare September 3, 2019 19:17

steveloughran commented Sep 10, 2019

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java Show resolved Hide resolved

steveloughran force-pushed the s3/HADOOP-16458-globber branch from 0ece51b to a58d260 Compare September 11, 2019 13:29

apache deleted a comment from hadoop-yetus Sep 12, 2019

steveloughran force-pushed the s3/HADOOP-16458-globber branch from a58d260 to d43dce6 Compare September 12, 2019 14:38

apache deleted a comment from hadoop-yetus Sep 16, 2019

steveloughran force-pushed the s3/HADOOP-16458-globber branch from d43dce6 to fc56e9e Compare September 27, 2019 17:06

steveloughran added 15 commits October 1, 2019 16:35

HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.

2139a5a

This adds the ITestS3AFSMainOperations test so that we have some globber tests against the S3A FS. They didn't find problems there, only a bug in setWorkingDir(). Change-Id: Idbdc4b049fa70fa31aa701b2e1c570c5bd208521

HADOOP-16458: Globber to support a no-symlink-FS option

241fa4b

this eliminates HEAD requests on scans where candidates are files Change-Id: I56971977e28a3adf09cae3df1c3d6e634cbc7295

HADOOP-16458 test to explore listFiles/listStatus/globStatus calls on…

0b28288

… no-read paths key point: LIST always works, but HEAD and GET fail; in auth mode many such failures get skipped. Change-Id: I66fc17021d5f2b4015c9afbb446df6cb4df0b9ce

HADOOP-16458 add test for no-read-access delete to round the sequence…

6d18b49

… off Change-Id: Ibe7f502a92c67040dbdba464e13751fdbe0db4f3

HADOOP-16458: remove unused ITest; break up ITestRestrictedReadAccess…

526e964

… method into pieces Change-Id: Ic65c49bd512f4b8f33c211b9e4924b203165619f

HADOOP-16458 final feedback from gabor

191ac65

Change-Id: I6937aa4c7de8adc135d9eb6bf99cca1b69773014

HADOOP-16458 and address an existing checkstyle issue on the LocatedF…

eab5a8a

…ileStatusFetcher Change-Id: I592cff79fce8680daee55bae983465919734a8f1

HADOOP-16458 gabor's comments about comments

779c001

Change-Id: I792c6187fdaaf6c981ac635ec9b527a925ad3d14

HADOOP-16458. Fix findbugs reported null deref

8ac342e

Change-Id: I3c1d64749107e6f69be76cdcb4eab724cdb1f9ba

HADOOP-16458 address my own review comments

c43f683

Change-Id: I80e5054ae2a5b7c7b616586f9882f07fcf0b8f92

HADOOP-16458 IDE warnings

b81aa7a

This is the first time anyone has edited the file for a few years; most of the warnings are unrelated to the patch. Change-Id: I429bc5f3c28d33f0321890487faa7ac2fd7a5ddd

steveloughran force-pushed the s3/HADOOP-16458-globber branch from fc56e9e to b81aa7a Compare October 1, 2019 16:59

steveloughran closed this Oct 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 #1160

HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 #1160

steveloughran commented Jul 25, 2019

steveloughran commented Jul 29, 2019

steveloughran Jul 30, 2019

bgaborg Jul 31, 2019

steveloughran Jul 31, 2019

bgaborg Jul 31, 2019

bgaborg Jul 31, 2019

steveloughran Sep 3, 2019

steveloughran Sep 3, 2019

bgaborg Jul 31, 2019

bgaborg commented Jul 31, 2019

steveloughran commented Jul 31, 2019

steveloughran commented Jul 31, 2019

bgaborg left a comment •

edited

Loading

steveloughran commented Aug 28, 2019

steveloughran commented Sep 3, 2019

hadoop-yetus commented Sep 4, 2019

hadoop-yetus commented Sep 12, 2019

hadoop-yetus commented Sep 27, 2019

steveloughran commented Oct 1, 2019

HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 #1160

HADOOP-16458 LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 #1160

Conversation

steveloughran commented Jul 25, 2019

steveloughran commented Jul 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgaborg commented Jul 31, 2019

steveloughran commented Jul 31, 2019

steveloughran commented Jul 31, 2019

bgaborg left a comment • edited Loading

Choose a reason for hiding this comment

steveloughran commented Aug 28, 2019

steveloughran commented Sep 3, 2019

hadoop-yetus commented Sep 4, 2019

hadoop-yetus commented Sep 12, 2019

hadoop-yetus commented Sep 27, 2019

steveloughran commented Oct 1, 2019

bgaborg left a comment •

edited

Loading