[Bug]: gradle check failing with java heap OutOfMemoryError, capturing heap dumps if available

[reta]

Signed-off-by: Andriy Redko [email protected]

Description

Capturing heap dumps if available in case build fails with OutOfMemoryError. We do run build steps with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=.../build/heapdump, for example:

[-Dgradle.dist.lib=/home/opensearch/.gradle/wrapper/dists/gradle-7.5.1-all/1ehga6e77gqps5uk2kc5kf1vc/gradle-7.5.1/lib, -Dgradle.user.home=/home/opensearch/.gradle, -Dgradle.worker.jar=/home/opensearch/.gradle/caches/7.5.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.hea
dless=true, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dtests.artifact=rest-api-spec, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=86BB942
FF1760033, -Dtests.task=:rest-api-spec:test, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -XX:HeapDumpPath=/home/opensearch/rest-api-spec/build/heapdump, -javaagent:../../tmp/expandedArchives/org.jacoco.agent-0.8.8.jar_a33b649e552c51298e5a242c2f0d0e3c/jacocoagent.jar=destfile=../../jacoco
/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx2g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/opensearch/rest-api-spec/build/testrun/test/temp, -Duser.country=CA, -Duser.language=en, -Duser.variant, -ea]  

Issues Resolved

Part of opensearch-project/OpenSearch#3973

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[reta]

@peterzhuamazon could you please take a look? thanks!

[peterzhuamazon]

@reta duplicates?
#2515

[peterzhuamazon]

And also in the PR I sent I set the file to specifically only archive during failure while yours use empty archive.
Not sure which implementation is better in this case.

Thanks.

[gaiksaya]

I think this will help to collect and compare logs when gradle check is successful and failed. So better to have the archive file for both scenarios since this is a recurring issue.

[peterzhuamazon]

I think this will help to collect and compare logs when gradle check is successful and failed. So better to have the archive file for both scenarios since this is a recurring issue.

The thing is the file does not even exist if there is no OOM.

[reta]

@peterzhuamazon sorry for duplication, have not seen one, I think using failure section would be appropriate here, as per your PR

[reta]

I think this will help to collect and compare logs when gradle check is successful and failed. So better to have the archive file for both scenarios since this is a recurring issue.

The thing is the file does not even exist if there is no OOM.

Correct, I belive allowEmptyArchive should address that (and no fail the buid step)

[reta]

@reta duplicates? #2515

My bad :(

[codecov-commenter]

Codecov Report

Merging #2523 (63da90e) into main (694a8b0) will increase coverage by 5.60%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #2523      +/-   ##
=============================================
+ Coverage     94.39%   100.00%   +5.60%     
=============================================
  Files           219         6     -213     
  Lines          4462       105    -4357     
  Branches         29        19      -10     
=============================================
- Hits           4212       105    -4107     
+ Misses          244         0     -244     
+ Partials          6         0       -6     

Impacted Files	Coverage Δ
src/ci_workflow/ci_manifests.py
src/system/working_directory.py
src/test_workflow/test_cluster.py
src/run_manifests.py
src/jenkins/TestManifest.groovy
src/system/config_file.py
tests/jenkins/jobs/BuildAssembleUpload_Jenkinsfile
src/system/os.py
src/ci_workflow/ci_check_gradle_dependencies.py
...ts_workflow/component_opensearch_dashboards_min.py
... and 203 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[peterzhuamazon]

@reta any issue for me to close this PR in favor of #2515?

[reta]

@reta any issue for me to close this PR in favor of #2515?

Closing it, sorry about that @peterzhuamazon