Add search request timeouts for correlations workflows

[goyamegh]

Description

Add search request timeouts to close hanging tasks for correlations waiting on searches.

Issues Resolved

#879

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 57 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@689760e). Click here to learn what that means.
Report is 1 commits behind head on main.

Files	Patch %	Lines
...ics/transport/TransportCorrelateFindingAction.java	0.00%	37 Missing ⚠️
...yanalytics/correlation/VectorEmbeddingsEngine.java	0.00%	14 Missing ⚠️
...arch/securityanalytics/correlation/JoinEngine.java	0.00%	6 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #893   +/-   ##
=======================================
  Coverage        ?   25.01%           
  Complexity      ?     1047           
=======================================
  Files           ?      277           
  Lines           ?    12771           
  Branches        ?     1391           
=======================================
  Hits            ?     3195           
  Misses          ?     9304           
  Partials        ?      272           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eirsep]

Plz remove perf_debug prefix in all logs

[goyamegh]

Fixed logging.

[eirsep]

Lgtm
Minor comments

[eirsep]

Can log message convey that search timed out hence indexing failed

[goyamegh]

Fixed the messaging to include the response itself as timeout can be one of the responses.

[eirsep]

Why such an aggressive timeout

[goyamegh]

We were seeing correlations tasks hanging around due to searches. Since these are system indices, we have configured a standard 10 second timeout for now, but we can discuss and make it configurable in future.

[eirsep]

Empty check?(i see u didnt change the code but better to add)

[goyamegh]

I added this check intentionally as I was observing that the flow was continuing even when we throw the exception in the if block above. So, putting this in the else block will avoid the ArrayOutOfBoundsException then.

[eirsep]

typically timeout should be a setting but we can do a quick follow up for that

[opensearch-trigger-bot]

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.11 2.11
# Navigate to the new working tree
cd .worktrees/backport-2.11
# Create a new branch
git switch --create backport/backport-893-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 75c442902f2715e4b41f96e77230ac773d0eb2d2
# Push it to GitHub
git push --set-upstream origin backport/backport-893-to-2.11
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport/backport-893-to-2.11.

[opensearch-trigger-bot]

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.7 2.7
# Navigate to the new working tree
cd .worktrees/backport-2.7
# Create a new branch
git switch --create backport/backport-893-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 75c442902f2715e4b41f96e77230ac773d0eb2d2
# Push it to GitHub
git push --set-upstream origin backport/backport-893-to-2.7
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport/backport-893-to-2.7.

[opensearch-trigger-bot]

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.9 2.9
# Navigate to the new working tree
cd .worktrees/backport-2.9
# Create a new branch
git switch --create backport/backport-893-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 75c442902f2715e4b41f96e77230ac773d0eb2d2
# Push it to GitHub
git push --set-upstream origin backport/backport-893-to-2.9
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport/backport-893-to-2.9.