Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[http_endpoint input] Add support for including headers and preserving original events #26279

Merged
merged 13 commits into from
Jun 22, 2021

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented Jun 14, 2021

What does this PR do?

This adds the possibility to give a list of headers that, if they exist in the incoming request, will be added as a separate object. Also adds the possibility to preserve the raw event as part of event.original.

Example output here:

Config example:

- type: http_endpoint
  enabled: true
  listen_address: 10.10.10.10
  listen_port: 8080
  preserve_original_event: true
  include_headers: ["TestHeader"]
  "json": {
    "test": "somevalue"
  },
  "event": {
    "original": "{\"test\":\"somevalue\"}"
  },
  "headers": {
    "TestHeader": "test"
  }

Why is it important?

Certain modules is hard to parse data without the extra details from the headers, github webhooks is one example of that, in which the data type is stored in the header

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 14, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 14, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26279 updated

  • Start Time: 2021-06-22T10:29:15.458+0000

  • Duration: 101 min 44 sec

  • Commit: fd4ef01

Test stats 🧪

Test Results
Failed 0
Passed 7385
Skipped 1201
Total 8586

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7385
Skipped 1201
Total 8586

@mergify
Copy link
Contributor

mergify bot commented Jun 14, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpendpoint_header_original upstream/httpendpoint_header_original
git merge upstream/master
git push upstream httpendpoint_header_original

x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/http_endpoint/config.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/http_endpoint/handler.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/http_endpoint/handler.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/http_endpoint/handler.go Outdated Show resolved Hide resolved
x-pack/filebeat/tests/system/test_http_endpoint.py Outdated Show resolved Hide resolved
x-pack/filebeat/tests/system/test_http_endpoint.py Outdated Show resolved Hide resolved
@P1llus P1llus requested a review from andrewkroh June 22, 2021 12:45
@P1llus P1llus merged commit 794df17 into elastic:master Jun 22, 2021
@P1llus P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 22, 2021
mergify bot pushed a commit that referenced this pull request Jun 22, 2021
…g original events (#26279)

* adding header support and event.original for the http_endpoint input

* add changelog

* updating docs

* adding tests

* applying comments from PR

* adding more tests, and applying changes related to PR comments

* linting

* stashing changes

* small changes

* Linting

* adding comments from PR

(cherry picked from commit 794df17)
P1llus added a commit that referenced this pull request Jun 23, 2021
…g original events (#26279) (#26417)

* adding header support and event.original for the http_endpoint input

* add changelog

* updating docs

* adding tests

* applying comments from PR

* adding more tests, and applying changes related to PR comments

* linting

* stashing changes

* small changes

* Linting

* adding comments from PR

(cherry picked from commit 794df17)

Co-authored-by: Marius Iversen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
7.14 Candidate backport-v7.14.0 Automated backport with mergify Filebeat Filebeat
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants