Capture memory leak failures during tests

[peternied]

Description

Tests will automatically save memory dumps when out of memory issues
occur in java test cases. Updated CI to also pick these up for
troubleshooting as needed.

Issues Resolved

• Resolves Memory leak on gradlew tests https://opensearch.atlassian.net/browse/MIGRATIONS-2250

Testing

Ran a previous iteration with low memory cap and inspected the assets captured. Hprof file zip where the memory limit was artificially lowered to force OOMs, workflow link.

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.91%. Comparing base (fc62f57) to head (35c77f9).
Report is 66 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1188      +/-   ##
============================================
- Coverage     80.93%   80.91%   -0.02%     
- Complexity     2995     2997       +2     
============================================
  Files           407      407              
  Lines         15241    15241              
  Branches       1021     1021              
============================================
- Hits          12336    12333       -3     
- Misses         2277     2284       +7     
+ Partials        628      624       -4     

Flag	Coverage Δ
unittests	80.91% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gregschohn]

In the test section, please point to the build that has the hprof files.
How long do we keep these (or any) artifacts for? If that window isn't very long, maybe some screen shots would be nice to show others.

11:07
How long do we keep those files (or any artifacts) around for?

[peternied]

@gregschohn I've updated the description to include an example:

Ran a previous iteration with low memory cap and inspected the assets captured. Hprof file zip where the memory limit was artificially lowered to force OOMs, workflow link.

---

How long do we keep those files (or any artifacts) around for?

They are around for the default period 90 days - not intented to be long term shortage, but I'd expect more than enough for a one-off issue that we drive to resolution.

[gregschohn]

Thanks Peter! Visually, one can find the artifacts for these hprof files in a build's summary at the bottom in the artifacts section

[gregschohn]

Thanks Peter. This is great - this could benefit users running gradle themselves too.