[Filebeat][Salesforce] Data duplication in Salesforce filebeat module

[yug-rajani]

When the Salesforce filebeat module is run, the data collected is duplicate because of the issue with the cursor set. The cursor should ideally be storing the created date for the last ingested event. However, after the update of the mechanism of chained calls as a part of this PR from the Elastic Team, the cursor value is not being set which leads to duplication of data being ingested because of the “WHERE” condition in the SOQL Query not running with the expected cursor value.

• Version: 8.4.1
• Operating System: CentOS 7
• Related Issue: [Filebeat][httpjson] Value of cursor is not being set in the first call in the chained calls #33213
• Steps to Reproduce:
  • Build filebeat inside x-pack/filebeat using mage build
  • Enable salesforce filebeat module
  • Configure the module and filebeat.yml to add the required credentials
  • Run filebeat
  • The data ingested is duplicate at every interval. For example, if an interval of 5 mins is set, the same set of events would be ingested every 5 mins.
• Root Cause: The value of the cursor last_published_login is not being set. This seems to be a limitation in the implementation/enhancement of the chained calls mechanism in httpjson.
• PR Link (with breaking changes): [filebeat][httpjson][bugfix] - root request pagination now working with chaining #32722