-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[filebeat][gcp-pubsub] - Set EventNormalization as false #36716
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
This pull request is now in conflicts. Could you fix it? 🙏
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
* Update EventNormalization as false for GCP Pubsub
Proposed commit message
EventNormalization is ON by default which iterates over all the fields for each event and if any nested fields are observed, it removes null values and drops unsupported datatypes.
However, just like AWS S3 Input, GCP PubSub input only produces events with basic types, hence EventNormalization is not required. This reduces the processing by 2 hops as seen from CPU Profile.
Performance Tests:
1.
With Event Normalization ~ 700 messages / sec
Without Event Normalization ~ 670 messages / sec
With Event Normalization ~ 1560 messages / sec
Without Event Normalization ~ 1600 messages / sec
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs