Adjust memory requests and limits for elastic-agent when run in Kubernetes cluster

[MichaelKatsoulis]

What does this PR do?

This PR adjusts the default memory requests and limits for elastic-agent when it runs in a Kubernetes cluster.
The new requests and limits will be

resources:
  limits:
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 500Mi

These will be adequate even in big load (tested with 95 pods per node).

Why is it important?

In latest elastic-agent versions the baseline memory consumption is higher, thus adding Kubernetes and system
integration leads to often OOM killed pods.
The previous memory limit set was 700Mb

After investigation with various number of pods we have seen that in ^8.15.* versions
the memory consumption can reach from 750 to 950 Mb.
This varies depending on the number of pods a single elastic agent has to monitor.
For example:

45 pods per node: mem consumption up to 810 MB
60  pods per node: mem consumption up to 830 Mb
75 pods per node: mem consumption up to 930 Mb
95 pods per node: mem consumption up to 990 MB

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

The inadequate memory limits lead to elastic-agent getting often OOM killed in newer versions.

How to test this PR locally

Deploy Elastic Agent with system and Kubernetes integration following the Kibana instructions and watch the
memory consumption of the pod with watch "kubectl top pod -n kube-system" and its status.

Related issues

• Relates Observability Kubernetes Onboarding doesn't ship data #5613
• Relates Kubernetes metadata overwhelms memory limits in the Agent process #4729

[mergify]

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[pkoutsovasilis]

Hey @MichaelKatsoulis 👋 could you check the failures in CI? I think that some other files also need the change of the limits

[MichaelKatsoulis]

@pchila do you have any idea why these build kite integrations tests are failing? It is not obvious to me.
The only updates to this PR are the slight increase of memory requests and limits in elastic agent in Kubernetes.

[pchila]

@MichaelKatsoulis Not sure why I got pulled in this PR as there are already a lot of knowledgeable people involved, the reason for the integration tests failure on this PR is that those same tests are currently broken also on main and investigation is ongoing. You should update your branch once main is green again and check the outcome of those tests again.

If your PR has sensitive time constraints so that you need to merge it before main becomes green again I guess you can present your case to one of the repo admins for a force merge

[MichaelKatsoulis]

@pchila I pulled you in cause it seemed to me that you have worked a lot with those failing integrations tests. Thanks for your time and information.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube