Cherry-pick #22320 to 7.x: [filebeat][httpjson] Initial implementation of httpjson v2

[marc-gr]

Cherry-pick of PR #22320 to 7.x branch. Original message:

What does this PR do?

Creates an initial implementation of the new Httpjson input.

Why is it important?

HTTPJSON is too specific to our particular modules use cases and is hard to extend or reuse out of them. For this we wanted to do an effort to generalize the input so it can be used in a more flexible way.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• The possible values that a transform modifies are defined by where it is configured
• Load cursor from registry
• Store cursor in registry
• Custom cursor values
• Supports oauth
• Supports basic auth
• Supports automatic retries
• Rate limit with custom values
• Values can have a default in case of empty or failure
• Value templates use predefined helper functions
  • now
  • parseDuration
  • parseDate
  • formatDate
  • parseTimestamp
  • parseTimestampMilli
  • parseTimestamlNano
  • getRFC5988Link
• Request transforms
  • Set
  • Append
  • Delete
• Response transforms
  • Set
  • Append
  • Delete
  • Split
    • Arrays
    • Maps
• Response pagination
  • Set
  • Append
  • Delete
• Redirects can forward headers
• Documentation for the new input
• Update old documentation to mention deprecation and migration steps
• Add missing tests

How to test this PR locally

• Set a mock endpoint to respond to http requests
• Set up a config that uses the new input

- type: httpjson
  is_v2: true
  interval: 10s
  request.method: post
  request.url: https://example.com
  request.transforms:
    - set:
        target: body.foo
        value: bazz
  response.transforms:
    - delete:
        target: body.this_is_private
  response.pagination:
    - set:
        target: url.value
        value: "https://example.com/page"
    - set:
        target: url.params.p
        value: "{{.last_response.body.page}}"

output.console:
  pretty: true

Then in x-pack/filebeat run mage build && ./filebeat -e -v and check the output.

Related issues

• Closes Add support into the httpjson input to collect values of maps in different events #21465
• Closes Support HTTP query parameters in httpjson filebeat input #20559
• Closes [Filebeat][httpjson] Add cursor support #19486
• Relates [Filebeat][httpjson] Substitute custom rate limit implementation for a third party #20379

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #22920 updated

• Start Time: 2020-12-09T08:48:06.836+0000

• Duration: 55 min 42 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2411
Skipped	259
Total	2670

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2411
Skipped	259
Total	2670