Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-17605. Reduce memory overhead of TestBPOfferService #6996

Merged
merged 1 commit into from
Aug 19, 2024

Conversation

sodonnel
Copy link
Contributor

Description of PR

With default memory settings TestBPOfferService often errors with GC overhead / OOM errors. After analysing the heap dump, this is due to Mockito holding lots of references to invocations on a datanode mock.

This mock is never checked for invocations so we can mark it as stubonly and allow Mockito to drop the invocations. This small change allows the test to consistently pass on my local environment.

How was this patch tested?

This is a test only change. I ran the test several times in my fork to ensure it passed consistently when it often failed prior to the fix.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 45m 18s trunk passed
+1 💚 compile 1m 25s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 compile 1m 20s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 checkstyle 1m 11s trunk passed
+1 💚 mvnsite 1m 27s trunk passed
+1 💚 javadoc 1m 11s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 1m 40s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 spotbugs 3m 35s trunk passed
+1 💚 shadedclient 38m 27s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 15s the patch passed
+1 💚 compile 1m 18s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javac 1m 18s the patch passed
+1 💚 compile 1m 10s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 javac 1m 10s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 58s the patch passed
+1 💚 mvnsite 1m 12s the patch passed
+1 💚 javadoc 0m 54s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 1m 33s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 spotbugs 3m 23s the patch passed
+1 💚 shadedclient 39m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 234m 9s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 48s The patch does not generate ASF License warnings.
380m 0s
Reason Tests
Failed junit tests hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
hadoop.hdfs.TestErasureCodingPolicies
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6996/1/artifact/out/Dockerfile
GITHUB PR #6996
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 744b31aeaa82 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9114fed
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6996/1/testReport/
Max. process+thread count 3553 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6996/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@sodonnel
Copy link
Contributor Author

Test failures are unrelated. @umamaheswararao could you give this a review if you get a minute? Its a one line change in a test file only. Thanks!

@slfan1989
Copy link
Contributor

LGTM. This modification is reasonable.

@sodonnel sodonnel merged commit df08e0d into apache:trunk Aug 19, 2024
1 of 4 checks passed
@sodonnel
Copy link
Contributor Author

Thanks for the review @slfan1989

KeeProMise pushed a commit to KeeProMise/hadoop that referenced this pull request Sep 9, 2024
Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants