Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator #3331

Merged
merged 5 commits into from
Nov 5, 2021

Conversation

sumangala-patki
Copy link
Contributor

@sumangala-patki sumangala-patki commented Aug 25, 2021

Hadoop yetus run shows a java compiler warning for unchecked casting of Object to Iterator<FileStatus> in a method of AbfsListStatusRemoteIterator class. This is resolved by introducing a new class to hold the iterator and exception thrown when applicable.

This PR also adds a potential fix and enables logging to facilitate better investigation of the occasional transient failure in the corresponding test class, tracked in HADOOP-17797. The additional logging introduced here affects only the test class for this functionality.

@sumangala-patki
Copy link
Contributor Author

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

AppendBlob HNS OAuth

[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 1
[ERROR] Failures: 
[ERROR]   ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166 » IO
[ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut
[ERROR]   ITestSmallWriteOptimization>AbstractAbfsScaleTest.setup:49->AbstractAbfsIntegrationTest.setup:174->AbstractAbfsIntegrationTest.createFileSystem:303 » FileAlreadyExists 
[ERROR] Tests run: 259, Failures: 0, Errors: 2, Skipped: 75

HNS OAuth

[ERROR] Failures: 
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 14 is not within the expected range: [5.60, 8.40].
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 1
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 98
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testRenameDirectoryMoveToExistingDirectory:526 » TestTimedOut
[ERROR] Tests run: 259, Failures: 0, Errors: 1, Skipped: 52

HNS SharedKey

[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 54
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

NonHNS SharedKey

[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Failures: 
[ERROR]   ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:93->testWithDifferentCustomTokenFetchRetry:122->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0)
[ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 276
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

Copy link
Contributor

@snvijaya snvijaya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

production code looks good.

test wise

  • too much duplication going in: makes maintenance harder. please refactor both the remove and the assert on fileNames size
  • lots of log @ debug/trace. Are these actual failures? If so they should be assertions. If not, but still a sign of recurrent issues: log at debug. And explain why it happens/why it is harmless.

@sumangala-patki
Copy link
Contributor Author

Hi @steveloughran, thanks for the review. The transient failure in HADOOP-17797 has been observed in each of these tests: testWithAbfsIteratorDisabled, testWithAbfsIteratorDisabledWithoutHasNext, testAbfsIteratorWithoutHasNext
The failure seems to be associated with the list of fileNames that is populated by the executor task on creating test files. Was able to reproduce it a few times, and found that failure always results from this list missing a filename. Have added an assert to verify addition to list whenever a test file is created. The failure seems to stem from concurrency-related issues within the test as opposed to list iterator functionality, so the initial asserts should help us distinguish these from actual listing errors

@sumangala-patki sumangala-patki marked this pull request as ready for review August 29, 2021 12:02
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 46s trunk passed
+1 💚 compile 0m 40s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 0m 36s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 0m 28s trunk passed
+1 💚 mvnsite 0m 41s trunk passed
+1 💚 javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 32s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 1m 2s trunk passed
+1 💚 shadedclient 14m 13s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 30s the patch passed
+1 💚 compile 0m 30s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌ javac 0m 30s /results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 24 unchanged - 1 fixed = 25 total (was 25)
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 0m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 18s the patch passed
+1 💚 mvnsite 0m 31s the patch passed
+1 💚 javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 21s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 1m 2s the patch passed
+1 💚 shadedclient 13m 58s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 10s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
83m 47s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/5/artifact/out/Dockerfile
GITHUB PR #3331
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux f4b40f90f9c8 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 5b9e041
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/5/testReport/
Max. process+thread count 547 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Sep 6, 2021
@apache apache deleted a comment from hadoop-yetus Sep 6, 2021
@apache apache deleted a comment from hadoop-yetus Sep 6, 2021
@apache apache deleted a comment from hadoop-yetus Sep 6, 2021
@sumangala-patki
Copy link
Contributor Author

Yetus error is a WASB side compiler warning tracked in JIRA, and does not involve ABFS code.

@steveloughran steveloughran merged commit 19644c0 into apache:trunk Nov 5, 2021
@steveloughran
Copy link
Contributor

merged into trunk; if you can verify a test run of this cherrypicked works on branch 3.3, i will add there too.

thanks.
ps: dont feel shy about reminding me of things to review; it's easy to forget about these small patches, but they keep the whole codebase healthy

asfgit pushed a commit that referenced this pull request Mar 1, 2022
…tatusRemoteIterator (#3331)

closes #3331

Contributed by Sumangala Patki

Change-Id: I6cca91c8bcc34052c5233035f14a576f23086067
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
…tatusRemoteIterator (apache#3331)


closes apache#3331 

Contributed by Sumangala Patki
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants