Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate disable event normalization processing in Cloudfoundry input #34407

Open
MichaelKatsoulis opened this issue Jan 27, 2023 · 1 comment
Assignees
Labels
investigate Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team

Comments

@MichaelKatsoulis
Copy link
Contributor

In this issue we will evaluate the benefits in cpu utilisation of filebeat cloud foundry input if we disable event normalization to reduce allocations when processing events.
This was made possible due to this PR.

The tests are performed with latest Elasticsearch 8.6 deployed on Elastic Cloud.
The https://apps.sys.giz-2.cf-obs.elastic.dev PCF cluster will be used and traffic will be generated using a log-generator app, each instance of which generates 100K events per minute.
There will be a comparison of cpu usage between a filebeat that disables event normalization and the default filebeat.

@MichaelKatsoulis MichaelKatsoulis self-assigned this Jan 27, 2023
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 27, 2023
@MichaelKatsoulis MichaelKatsoulis added Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 27, 2023
@MichaelKatsoulis
Copy link
Contributor Author

MichaelKatsoulis commented Jan 27, 2023

Filebeat in all cases is deployed as an app on PCF with Memory Allocated 1 GB. Due to each Diego cell capacity of 64GB ram and 16 cpus, this translates to Filebeat entitlement of (1/64)*16 = 0.25 cpus per instance.

A is default filebeat and B filebeat with normalization disabled. In each case filebeat run for 30 minutes and we collected the average cpu utilisation of all instances during that period.

Test 1
900K events per minute and 8 filebeat instances deployed.
A. cf cpu shows average 148% of entitlement. This translates to 0,37 cpus
B. cf cpu shows average 137,5% of entitlement. This translates to 0,34 cpus

Verdict: Filebeat with normalization disabled utilises ~8% less cpu

Test 2
1M events per minute and 8 filebeat instances deployed.
A. cf cpu shows average 162% of entitlement. This translates to 0,405 cpus
B. cf cpu shows average 146% of entitlement. This translates to 0,365 cpus

Verdict: Filebeat with normalization disabled utilises ~10% less cpu

I was unable to add more traffic to the cluster. Maybe I hit a bottleneck at Elasticsearch. But the sample is good and shows decrease in cpu utilisation.

Next step:
Investigate how events stored in Elasticsearch are affected by disabling normalization.

cc @gizas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
investigate Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team
Projects
None yet
Development

No branches or pull requests

1 participant