[ML] APM Correlations: Fix usage in load balancing/HA setups.

[walterra]

Summary

Part of #114046.

• The way we customized the use of search strategies caused issues with race conditions when multiple Kibana instances were used for load balancing. This PR migrates away from search strategies and uses regular APM API endpoints.
• The task that manages calling the sequence of queries to run the correlations analysis is now in a custom React hook (useFailedTransactionsCorrelations / useLatencyCorrelations) instead of a task on the Kibana server side. While they show up as new lines/files in the git diff, the code for the hooks is more or less a combination of the previous useSearchStrategy and the server side service files that managed queries and state.
• The consuming React UI components only needed minimal changes. The above mentioned hooks return the same data structure as the previously used useSearchStrategy. This also means functional UI tests didn't need any changes and should pass as is.
• API integration tests have been added for the individual new endpoints. The test files that were previously used for the search strategies are still there to simulate a full analysis run, the assertions for the resulting data have the same values, it's just the structure that had to be adapted.
• Previously all ES queries of the analysis were run sequentially. The new endpoints run ES queries in parallel where possible. Chunking is managed in the hooks on the client side.
• For now the endpoints use the standard current user's esClient. I tried to use the APM client, but it was missing a wrapper for the fieldCaps method and I ran into a problem when trying to construct a random_score query. Sticking to the esClient allowed to leave most of the functions that run the actual queries unchanged. If possible I'd like to pick this up in a follow up. All the endpoints still use withApmSpan() now though.
• The previous use of generators was also refactored away, as mentioned above, the queries are now run in parallel.
• Because we might run up to hundreds of similar requests for correlation analysis, we don't want the analysis to fail if just a single query fails like we did in the previous search strategy based task. I created a util splitAllSettledPromises() to handle Promise.allSettled() and split the results and errors to make the handling easier. Better naming suggestions are welcome 😅 . A future improvement could be to not run individual queries but combine them into nested aggs or using msearch. That's out of scope for this PR though.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for breaking API changes and was labeled appropriately

[dgieselaar]

My feedback mostly revolves around parallelising some of the requests which can hopefully improve performance. Long term I think we should move this to one or two API calls and remove progressive loading, I don't think it's worth the complexity.

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9872afc
• Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
apm	1184	1189	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
apm	2.7MB	2.7MB	+4.0KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
apm	37	41	+4

History

• 💔 Build #4943 failed 22f785c
• 💔 Build #4910 failed de63d24
• 💔 Build #4767 failed 86b78c1
• 💔 Build #4761 failed 24d1d2b
• 💚 Build #4618 succeeded 97890cf
• 💚 Build #4185 succeeded 5310686

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

[dgieselaar]

This can be parallelised as well no?

[walterra]

I parallelised the first two requests to get the chart data first so we can show the chart as soon as it's available. This one is the first of the requests of the analysis, the requests after that one depends on its output.

[dgieselaar]

I'm not sure if I fully understand, but there's no need (AFAICT) to delay starting the request until the data for the chart has been fully loaded. E.g. you can start the request but only await it once the data for the charts have loaded.

[dgieselaar]

This too?

[walterra]

This splits all field candidates into chunks, the chunks are called in sequence here on the client side, but all field candidates of a chunk are then queried in parallel on the Kibana server side.

[dgieselaar]

Yes, but why call these in sequence? how many blocking calls can we expect here?

[walterra]

This was made in the spirit of "make it slow". I'm sure this can be further optimized, in this PR we started to parallelize the server side calls and play it safe on the client side. Since the field candidates and field value pairs are generated dynamically, we don't want to allow to run an unlimited amount of queries in parallel. Field candidates are usually in the dozens, field value pairs can be in the hundreds.

[dgieselaar]

I don't see how running dozens of requests sequentially is a better experience. We can use pLimit here to limit the number of concurrent requests. Something like 5 sounds like a good start.

[sorenlouv]

+1 for using p-limit here

[walterra]

p-limit sounds like a good idea worth pursuing but can we agree to do this in a follow up?

[dgieselaar]

yes that's fine 👍

[dgieselaar]

I'll approve this with the caveat that I haven't fully tested it - I'm mostly new to this code and we don't have a much time, and this bug fix is sorely needed. Thanks @walterra!