[Filebeat][httpjson] httpjson chain calls #29816

kush-elastic · 2022-01-12T13:34:23Z

Enhancement

What does this PR do?

This PR contains code for extending httpjson to run chain calls and collect information from multiple urls.
This httpjson code enables users to perform chains in a single manner.

Why is it important?

Current filebeat httpjson input doesn't support chain calls.
For Example:
Salesforce: To collect eventLogFile from salesforce, we first need to collect logfile ids from Salesforce REST API and then use those logfile ids to collect more information from Salesforce.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Check with Salesforce REST API to collect EventLogFile.

Related issues

Relates [Filebeat][httpjson input] Possibility to chain requests #22590

Use cases

Configure chain parameters with each chain in step parameter.

mergify · 2022-01-12T13:34:55Z

This pull request does not have a backport label. Could you fix it @kush-elastic? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-01-12T13:49:39Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-29T09:43:29.351+0000
Duration: 59 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2044
Skipped	159
Total	2203

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2022-01-12T13:54:25Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_httpjson_chain_calls upstream/salesforce_httpjson_chain_calls
git merge upstream/master
git push upstream salesforce_httpjson_chain_calls

elasticmachine · 2022-01-12T14:47:19Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2022-01-12T14:47:24Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

botelastic · 2022-01-12T14:47:26Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2022-01-18T14:12:47Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

efd6 · 2022-03-29T09:04:44Z

x-pack/filebeat/docs/inputs/input-httpjson.asciidoc

@@ -976,9 +1117,18 @@ image:images/input-httpjson-lifecycle.png[Request lifecycle]
 . The resulting transformed request is executed.
 . The server responds (here is where any retry or rate limit policy takes place when configured).
 . The response is transformed using the configured `response.transforms` and `response.split`.
+. If chain step is configured. step will generate new requests based on collected ids from responses. the requests will be transformed using configured `request.transforms`. resulting generated transformed requests will be executed. this process will happen for all the steps mentioned in chain.


Suggested change

. If chain step is configured. step will generate new requests based on collected ids from responses. the requests will be transformed using configured `request.transforms`. resulting generated transformed requests will be executed. this process will happen for all the steps mentioned in chain.

. If a chain step is configured. Each step will generate new requests based on collected IDs from responses. The requests will be transformed using configured `request.transforms` and the resulting generated transformed requests will be executed. This process will happen for all the steps mentioned in the chain.

efd6 · 2022-03-29T09:05:49Z

x-pack/filebeat/docs/inputs/input-httpjson.asciidoc

+. go back to step-2 for the next step.
+. publish collected responses from the last chain step.


Suggested change

. go back to step-2 for the next step.

. publish collected responses from the last chain step.

. Go back to step-2 for the next step.

. Publish collected responses from the last chain step.

efd6 · 2022-03-29T09:09:00Z

x-pack/filebeat/input/httpjson/input_test.go

+			setupServer: newChainTestServer(httptest.NewServer),
+			baseConfig: map[string]interface{}{
+				"interval":       10,
+				"request.method": "GET",


These instances of "GET" should also be changed to http.MethodGet (throughout this file).

efd6

LGTM, but please wait for others.

marc-gr · 2022-03-29T09:42:52Z

/test

* [filebeat][httpjson] Extend filebeat httpjson input for chained calls * Updates based on comments * Update on httpjson and nits * Resolve comments from Tiago * address comments and improved json_parser * Add entry to CHANGELOG.next.asciidoc * mage update and mage check for ci linting * address comments * Addressed comments * remove else as we have continue, close resp.Body in id collection mechanism and update json_parser to convert any type values to string * nits * mage check * Changes requested from elastic team * requested changes on logging level * requested changes for ids collection * Requested changes on input-httpjson.asciidoc and added tests for chain in input_test.go * Added regular tests with chain * Changes based on CI * Add New diagram for chain feature and updates based on CI changes * CI linter problem * removed unnecessary ci changes * golangci eventsCh->events * ignore few CI errors * update GET to http.MethodGet * Review changes- GET->http.MethodGet and nits (cherry picked from commit 2ec07f5) # Conflicts: # x-pack/filebeat/input/httpjson/pagination.go # x-pack/filebeat/input/httpjson/request.go # x-pack/filebeat/input/httpjson/response.go

rdner · 2022-03-29T12:34:08Z

@mtojek merging this PR might have broken the documentation, see https://elasticsearch-ci.elastic.co/blue/organizations/jenkins/elastic%2Bbeats%2Bpull-request%2Bbuild-docs/detail/elastic%20beats%20pull-request%20build-docs/13929/pipeline/

it's hard to tell the exact cause but at least I see these warnings

INFO:build_docs:asciidoctor: WARNING: ../../x-pack/filebeat/docs/inputs/input-httpjson.asciidoc: line 1042: id assigned to block already in use: response-json1
INFO:build_docs:asciidoctor: WARNING: invalid reference: response-json2

szabosteve · 2022-03-29T12:51:25Z

Hi @mtojek, @kush-elastic,

The first issue is caused by this line: https://github.com/elastic/beats/pull/29816/files#diff-531162d76f10990e57691d2298a46f2aa2a4e576a8ec0d17dd1a37c35303983fR1040 (as the ID is already in use).
The second one is caused by this: https://github.com/elastic/beats/pull/29816/files#diff-531162d76f10990e57691d2298a46f2aa2a4e576a8ec0d17dd1a37c35303983fR977 (response-json2 is an invalid reference)

ruflin · 2022-03-30T06:55:16Z

@mtojek Why is this planned to be backported to 8.2? This is not a bug fix.

mtojek · 2022-03-30T07:10:46Z

@ruflin isn't after the feature freeze now?

dedemorton · 2022-03-31T21:36:45Z

Sorry, I missed the review because I was on PTO. :/ Let's not worry too much about the documentation structure right now. We'll be moving to a new system and need to do some refactoring/cleanup as part of the migration, so let's handle layout issues then.

* [filebeat][httpjson] Extend filebeat httpjson input for chained calls * Updates based on comments * Update on httpjson and nits * Resolve comments from Tiago * address comments and improved json_parser * Add entry to CHANGELOG.next.asciidoc * mage update and mage check for ci linting * address comments * Addressed comments * remove else as we have continue, close resp.Body in id collection mechanism and update json_parser to convert any type values to string * nits * mage check * Changes requested from elastic team * requested changes on logging level * requested changes for ids collection * Requested changes on input-httpjson.asciidoc and added tests for chain in input_test.go * Added regular tests with chain * Changes based on CI * Add New diagram for chain feature and updates based on CI changes * CI linter problem * removed unnecessary ci changes * golangci eventsCh->events * ignore few CI errors * update GET to http.MethodGet * Review changes- GET->http.MethodGet and nits

kush-elastic added 5 commits December 8, 2021 16:55

[filebeat][httpjson] Extend filebeat httpjson input for chained calls

07e006a

Updates based on comments

d062e26

Update on httpjson and nits

664d95b

Resolve comments from Tiago

e288c0d

address comments and improved json_parser

27e71a2

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 12, 2022

kush-elastic requested a review from mtojek January 12, 2022 13:34

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 12, 2022

kush-elastic requested a review from belimawr January 12, 2022 13:35

kush-elastic self-assigned this Jan 12, 2022

kush-elastic added the enhancement label Jan 12, 2022

kush-elastic requested a review from marc-gr January 12, 2022 13:39

Add entry to CHANGELOG.next.asciidoc

24b1e87

Merge branch 'master' into salesforce_httpjson_chain_calls

28daafc

mtojek added the Team:Integrations Label for the Integrations team label Jan 12, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 12, 2022

mtojek added needs_team Indicates that the issue/PR needs a Team:* label Team:Security-External Integrations labels Jan 12, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 12, 2022

mage update and mage check for ci linting

915bc6d

kush-elastic changed the title ~~Salesforce httpjson chain calls~~ [Filebeat][httpjson] httpjson chain calls Jan 18, 2022

kush-elastic mentioned this pull request Jan 18, 2022

[filebeat][httpjson] Extend filebeat httpjson input for chained calls kush-elastic/beats#3

Closed

7 tasks

mtojek added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jan 18, 2022

kush-elastic added 2 commits March 29, 2022 12:18

ignore few CI errors

d55af74

update GET to http.MethodGet

a0a5d94

kush-elastic requested a review from a team as a code owner March 29, 2022 08:03

kush-elastic requested review from efd6 and removed request for a team March 29, 2022 08:04

mtojek requested a review from ruflin March 29, 2022 08:41

efd6 reviewed Mar 29, 2022

View reviewed changes

Review changes- GET->http.MethodGet and nits

6739df7

efd6 approved these changes Mar 29, 2022

View reviewed changes

mtojek approved these changes Mar 29, 2022

View reviewed changes

mtojek merged commit 2ec07f5 into elastic:main Mar 29, 2022

mergify bot mentioned this pull request Mar 29, 2022

[8.1](backport #29816) [Filebeat][httpjson] httpjson chain calls #31040

Closed

mtojek mentioned this pull request Mar 29, 2022

[DOCS] Fixes section ID #31046

Merged

mtojek added backport-v8.2.0 Automated backport with mergify and removed backport-v8.1.0 Automated backport with mergify labels Mar 29, 2022

ruflin removed the backport-v8.2.0 Automated backport with mergify label Mar 30, 2022

andrewkroh mentioned this pull request Mar 31, 2022

[Filebeat][httpjson input] Possibility to chain requests #22590

Closed

andrewkroh added the v8.3.0 label Mar 31, 2022

cmacknz mentioned this pull request Apr 4, 2022

[Filebeat][Inputs][HTTP JSON] Multi-Step/API Requests #31117

Closed

kush-elastic mentioned this pull request May 2, 2022

[Filebeat][Salesforce] Add Salesforce filebeat module #31486

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat][httpjson] httpjson chain calls #29816

[Filebeat][httpjson] httpjson chain calls #29816

kush-elastic commented Jan 12, 2022 •

edited

Loading

mergify bot commented Jan 12, 2022

elasticmachine commented Jan 12, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Jan 12, 2022

elasticmachine commented Jan 12, 2022

elasticmachine commented Jan 12, 2022

botelastic bot commented Jan 12, 2022

elasticmachine commented Jan 18, 2022

efd6 Mar 29, 2022

efd6 Mar 29, 2022

efd6 Mar 29, 2022

efd6 left a comment

marc-gr commented Mar 29, 2022

rdner commented Mar 29, 2022

szabosteve commented Mar 29, 2022 •

edited

Loading

ruflin commented Mar 30, 2022

mtojek commented Mar 30, 2022

dedemorton commented Mar 31, 2022 •

edited

Loading

	. If chain step is configured. step will generate new requests based on collected ids from responses. the requests will be transformed using configured `request.transforms`. resulting generated transformed requests will be executed. this process will happen for all the steps mentioned in chain.
	. If a chain step is configured. Each step will generate new requests based on collected IDs from responses. The requests will be transformed using configured `request.transforms` and the resulting generated transformed requests will be executed. This process will happen for all the steps mentioned in the chain.

		. go back to step-2 for the next step.
		. publish collected responses from the last chain step.

[Filebeat][httpjson] httpjson chain calls #29816

[Filebeat][httpjson] httpjson chain calls #29816

Conversation

kush-elastic commented Jan 12, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

Related issues

Use cases

mergify bot commented Jan 12, 2022

elasticmachine commented Jan 12, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Jan 12, 2022

elasticmachine commented Jan 12, 2022

elasticmachine commented Jan 12, 2022

botelastic bot commented Jan 12, 2022

elasticmachine commented Jan 18, 2022

efd6 Mar 29, 2022

Choose a reason for hiding this comment

efd6 Mar 29, 2022

Choose a reason for hiding this comment

efd6 Mar 29, 2022

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

marc-gr commented Mar 29, 2022

rdner commented Mar 29, 2022

szabosteve commented Mar 29, 2022 • edited Loading

ruflin commented Mar 30, 2022

mtojek commented Mar 30, 2022

dedemorton commented Mar 31, 2022 • edited Loading

kush-elastic commented Jan 12, 2022 •

edited

Loading

elasticmachine commented Jan 12, 2022 •

edited by jenkins-beats-ci bot

Loading

szabosteve commented Mar 29, 2022 •

edited

Loading

dedemorton commented Mar 31, 2022 •

edited

Loading