[Filebeat][httpjson] Value of cursor is not being set in the first call in the chained calls

[yug-rajani]

• Version: 8.4.1
• Operating System: CentOS 7
• Related Issue: [Filebeat][Salesforce] Data duplication in Salesforce filebeat module #33212
• PR Link (with breaking changes): [filebeat][httpjson][bugfix] - root request pagination now working with chaining #32722
• Reason:
  • This bug was found when running the Salesforce filebeat module. The value of the cursor was not set after the first call because of which the data is being duplicated.
• Probable Cause: if condition added as a part of this PR.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[ShourieG]

@yug-elastic thanks for bringing this up. Will be creating a PR to fix this soon. What's the urgency for this and what's the current impact ?

[yug-rajani]

@yug-elastic thanks for bringing this up. Will be creating a PR to fix this soon. What's the urgency for this and what's the current impact ?

@ShourieG, thanks for helping out!

The salesforce filebeat module is in production, which should be impacted by this, and the main issue that is being caused there is duplication of the ingested documents. I have created a separate issue to track it (#33212), but there might be other integrations/modules (leveraging httpjson chained calls mechanism) which might be impacted because of this issue.

As far as the urgency is concerned, @rameshelastic or @lalit-satapathy from the @elastic/obs-service-integrations team should be able to help you out with it based on their planning.

[ShourieG]

@yug-elastic can you provide the config for httpjson that is causing this behaviour. I've spoken to @lalit-satapathy on this, and reproducing the issue for httpjson would help us fix it.

[ShourieG]

@yug-elastic do you have any update on this ?

[kush-elastic]

Hey @ShourieG ,
Sorry for the delay. here is the httpjson config file for the chained calls where you can reproduce this same issue. let me know if you need anything apart from this.

filebeat.inputs:
- type: httpjson
  interval: 20s
  request.method: GET
  auth.oauth2:
    enabled: true
    client.id: {{ client_id }}
    client.secret: {{ client_secret }}
    token_url: {{ token_url }}
    user: {{ user }}
    password: {{ password }}
  request.url: {{ url }}/services/data/v52.0/query?q=login+rest
  request.proxy_url: {{ proxy_url }}
  request.transforms:
    - set:
        target: url.params.q
        value: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Login' AND CreatedDate > [[.cursor.last_published_login]] ORDER BY CreatedDate ASC NULLS FIRST"
        default: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Login' ORDER BY CreatedDate ASC NULLS FIRST"
  response.split:
    target: body.records
  chain:
    - step:
        request.url: {{ url }}/services/data/v52.0/sobjects/EventLogFile/$.records[:].Id/LogFile
        request.method: GET
        replace: $.records[:].Id
  cursor:
    last_published_login:
      value: '[[.last_event.CreatedDate]]'

[SpencerLN]

@ShourieG is there any chance of backporting this fix to 8.6?

[ShourieG]

@SpencerLN This has already been merged for the 8.6 release, so no issues.

[SpencerLN]

@ShourieG I could be missing something obvious here, but I don't see the commit in the 8.6 branch: https://github.com/elastic/beats/tree/8.6/x-pack/filebeat/input/httpjson