Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #20226 to 7.x: [Filebeat][httpjson] Convert httpjson input to a v2 input #20702

Merged
merged 1 commit into from
Aug 26, 2020

Conversation

marc-gr
Copy link
Contributor

@marc-gr marc-gr commented Aug 20, 2020

Cherry-pick of PR #20226 to 7.x branch. Original message:

What does this PR do?

Converts httpjson input to the new v2 stateless input.

  • Split old input in several new objects for easier understanding
  • Refactored go tests to simplify them and make them work with the new input

Why is it important?

Several modules rely on httpjson input, and would benefit a lot from the new v2 cursor feature that keeps persistence of a cursor between restarts.

This change moves httpjson input to new v2 input, preparing to be able to implement the cursor persistence state in a following change.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [] I have made corresponding changes to the documentation
    - [] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates to #19486

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2020
@marc-gr marc-gr force-pushed the backport_20226_7.x branch from c6729e9 to 891a01d Compare August 20, 2020 07:14
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 20, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20702 updated]

  • Start Time: 2020-08-26T09:45:37.137+0000

  • Duration: 54 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 2473
Skipped 388
Total 2861

@marc-gr
Copy link
Contributor Author

marc-gr commented Aug 20, 2020

This #20704 needs to be merged first

)

* Convert httpjson input to a v2 cursor input

* Add CHANGELOG entry

* Fix format errors

* Convert to stateless input and refactor:
- Paginator takes care of requesting next page info
- Rate limiter takes care of rate limiting requests
- Date cursor takes care of keeping track of cursor state

* Remove python tests

* Do not fail if there is no next page

* Refactor go integration tests to work with v2 input

* Do suggested changes to input and tests

* Update time.Periodic call with error return

* Change test duration values

* Sepparate sync test case

* Create custon url config type

* Change input.Run comment

* Change input.Run to only return on context cancellation

* Remove all usages of pkg/errors

(cherry picked from commit 929e838)
@marc-gr marc-gr force-pushed the backport_20226_7.x branch from 891a01d to fda732d Compare August 26, 2020 09:44
@marc-gr marc-gr merged commit 1d08e2a into elastic:7.x Aug 26, 2020
@marc-gr marc-gr deleted the backport_20226_7.x branch August 26, 2020 11:20
@zube zube bot removed the [zube]: Done label Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants