-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x-pack/filebeat/module/salesforce: Refactor a Salesforce module #37509
x-pack/filebeat/module/salesforce: Refactor a Salesforce module #37509
Conversation
- Update cursor to support custom fields from config - Update cursor implementation
…urations. for example, user will be able to collect login data from both event_log_file and real-time objects at the same time.
…c/beats into new_salesforce_input
…c/beats into new_salesforce_input
…c/beats into new_salesforce_input
…articular time in iteration so we wont face any memory or network spike
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
I don't know much about Salesforce, but glancing over the PR it looks correct.
Given the extensive size of the PR, reviewing all the changes thoroughly might pose a challenge. However, Kush and I have diligently gone through it, addressing numerous bugs along the way. @aliabbas-elastic and @niraj-elastic are helping in the multiple iterations of the review and testing process. Since this marks the initial alpha release, we have a window of opportunity to address any lingering bugs or implement enhancements. cc: @lalit-satapathy Will require your approval to merge this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving to un-gate alpha release, per FF.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving on testing side. Refer here for the testing comments
Thanks @aliabbas-elastic! |
Description
This pull request aims to restructure the existing filebeat Salesforce module, transitioning it to utilize the new Salesforce input. The changes are subject to review once the new Salesforce input has been merged.
The current implementation of the Salesforce module employs httpjson and cometd inputs to capture historical and real-time data from Salesforce, respectively. Historical data is obtained from EventLogFiles, which stores data in files that can be retrieved from Salesforce using REST API and SOQL Queries. We use httpjson input to download and process this historical data. Real-time data is collected through real-time event monitoring, utilizing the pub-sub mechanism supported by Salesforce and cometd input can be used for the same.
During the collection of data from historical forms, we encountered challenges, with a significant issue being out-of-memory (OOM) issues. These errors could be reproduced with a large number of files from Salesforce, causing the httpjson input to utilize a significant amount of memory and eventually get killed by the kernel (oomk). To address this issue, we propose a solution by introducing a new Salesforce input. This new input is easily triggerable and capable of managing a large volume of data.
We also have plans to enhance the current Salesforce module to support more features as part of this improvement. Since most issues were closely tied to Salesforce, the decision was made to create a new input rather than repeatedly addressing problems with the existing httpjson input.
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Configuration
Salesforce Module Configuration Parameters:
enabled
:true
var.api_version
:56
var.authentication
:enabled
: trueclient.id
: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"client.username
: "[email protected]"client.key_path
: client_key.pemurl
: https://login.salesforce.comClient Authentication Details:
enabled
: trueclient.id
: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"client.secret
: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"token_url
: "https://login.salesforce.com"username
: "[email protected]"password
: "P@$$W0₹D"var.url
:Event Log File Collection:
true
var.elf_interval
: 1 hourReal-Time Event Monitoring:
true
var.real_time_interval
: 5 minutesRelated issues