Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify kafka input to include parsing multiple messages #13965

Merged
merged 8 commits into from
Oct 9, 2019

Conversation

narph
Copy link
Contributor

@narph narph commented Oct 8, 2019

If the fileset using this input expects to receive multiple messages bundled under a specific field then the config option ExpandEventListFromField value can be assigned the name of the json group object.
For example in the case of azure fielsets the events are found under the json object "records".

 {
"records": [ {event1}, {event2}]
}

Initial draft PR #13776

@narph narph requested a review from faec October 8, 2019 14:59
@narph narph self-assigned this Oct 8, 2019
@narph narph added Team:Integrations Label for the Integrations team Filebeat Filebeat [zube]: In Review labels Oct 8, 2019
@narph narph requested a review from exekias October 8, 2019 15:00
filebeat/input/kafka/input.go Outdated Show resolved Hide resolved
CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
filebeat/input/kafka/input.go Show resolved Hide resolved
filebeat/input/kafka/config.go Outdated Show resolved Hide resolved
@narph narph requested a review from faec October 8, 2019 20:37
@narph
Copy link
Contributor Author

narph commented Oct 8, 2019

jenkins test this

Copy link
Contributor

@faec faec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation nits, otherwise looks good, thanks!

@@ -127,6 +127,17 @@ Kafka fetch settings:
*`max`*:: The maximum number of bytes to read per request. Defaults to 0
(no limit).

===== `expand_event_list_from_field`

If the fileset using this input expects to receive multiple messages bundled under a specific field then the config option `expand_event_list_from_field` value can be assigned the name of the json group object.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the name of the json group object" -> "the name of the field" is I think clearer about what value should go in the config

"fielsets" -> "filesets" in the following line

@narph narph merged commit 065f291 into elastic:master Oct 9, 2019
@narph narph deleted the kafka-azure branch October 9, 2019 08:58
@zube zube bot added [zube]: Done and removed [zube]: Inbox labels Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat Team:Integrations Label for the Integrations team v7.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants