[Elasticsearch Client] Limit Kibana's internal client's maxSockets

[afharo]

Summary

While testing #151110, we noticed that most of the issues come from a sudden spike in the generation of requests to ES.

While Kibana should aim to scale as needed for any incoming HTTP requests, its background tasks should be limited from branching too many ES connections at once. When this happens, it's usually a sign of non-scalable code running in the background.

This PR wants to test if limiting the number of background connections in the Kibana Internal Elasticsearch Client improves the overall performance experienced by users. The motivation is: limiting the resources for background processes leaves extra capacity to process requests from actual users.

A promising starting point #151110's scalability tests show a big improvement:

This PR	Baseline

FWIW, mission-critical background processes like Alerts can create their own client using core.elasticsearch.createClient() to circumvent this limitation if deemed necessary.

TODO:

• Test other scenarios apart from telemetry endpoints: https://buildkite.com/elastic/kibana-apis-capacity-testing/builds/301 TBD... waiting for @dmlemeshko to apply some changes to the runner so we can run from a branch
• Any improvements found in Single-User benchmarks? => Jobs: Roughly the same results.
• If agreed on this... document this behavior and scape-hatch for mission-critical apps.
• If we agree that creating too many sockets it's harmful to Kibana... should we revisit our default of Infinity even for user-triggered requests?

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risk Matrix

Risk	Probability	Severity	Mitigation/Notes
Some background processes may consume other mission-critical ones	High	High	It is happening today: while we set maxSockets: Infinity by default, high event loop delays might block all services. Also, the mission-critical ones can create custom clients to have their dedicated connection pool.

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 0b8a573

Metrics [docs]

✅ unchanged

History

• 💔 Build #109407 failed 8f9270666e19c3194b83a74329118450b82a33fa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[afharo]

After discussing this internally, we all agreed we'd better find a proper default to our current maxSockets value instead of adding a new one. Closing this PR in favour of #151911