Add new mode to multiline reader to aggregate constant number of lines #18352
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kvch
added
Filebeat
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
the
needs_team
💔 Build Failed
Expand to view the summary
Build stats
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
williamd67
reviewed
May 11, 2020
| if config.Type == patternMode || config.Type == nil { | ||
| return newMultilinePatternReader(r, separator, maxBytes, config) | ||
| } else if config.Type == countMode { | ||
| return newMultilineCountReader(r, separator, maxBytes, config) |
There was a problem hiding this comment.
I do not think it is needed to introduce a complete new reader. Some small modifications to the existing multline-reader will do exactly what is requested.
There was a problem hiding this comment.
It is not introducing a new type. Only the validation of the config options is separated.
williamd67
reviewed
May 11, 2020
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
7a3f738 to
6619106
Compare
andresrc
added
[zube]: Inbox
[zube]: In Review
and removed
[zube]: In Review
[zube]: Inbox
labels
|
Reading through the state machine I wonder if |
urso
reviewed
Jun 2, 2020
| @@ -163,6 +203,68 @@ func (mlr *Reader) readFirst() (reader.Message, error) { | |||
| } | |||
| } | |||
|
|
|||
| func (mlr *Reader) readFirstCount() (reader.Message, error) { | |||
There was a problem hiding this comment.
for the pattern matcher we need readFirst, because we don't want to validate against the line against the regular expression yet. But I don't think we need readFirstCount. We always check for the number of lines and create an event once the amount of lines have been reached.
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
8adf22b to
3592c83
Compare
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
urso
reviewed
Jun 9, 2020
urso
previously approved these changes
Jun 9, 2020
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
3592c83 to
eb91372
Compare
urso
reviewed
Jun 10, 2020
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
e4b0bcd to
914129f
Compare
urso
reviewed
Jun 15, 2020
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
06a04dd to
c6a5a66
Compare
urso
approved these changes
Jun 16, 2020
kvch
force-pushed
the
feature-libbeat-counter-multiline
branch
from
2eaa696 to
7da05b6
Compare
kvch
added a commit
to kvch/beats
that referenced
this pull request
Jun 17, 2020
elastic#18352) ## What does this PR do? This PR adds a new mode for the multiline reader of Libbeat (exposed in Filebeat). The new mode lets users to aggregate the configured number of lines into a single event. Example configuration to aggregate 5 lines: ```yaml muliline.type: count multiline.count_lines: 5 ``` This PR also adds a new configuration option `skip_newline`. If set, Filebeat does not add a newline when two events are concatenated. Closes elastic#18038 (cherry picked from commit e3f51ab)
6 tasks
kvch
added a commit
that referenced
this pull request
Jun 17, 2020
…ate constant number of lines (#19243) * Add new mode to multiline reader to aggregate constant number of lines (#18352) ## What does this PR do? This PR adds a new mode for the multiline reader of Libbeat (exposed in Filebeat). The new mode lets users to aggregate the configured number of lines into a single event. Example configuration to aggregate 5 lines: ```yaml muliline.type: count multiline.count_lines: 5 ``` This PR also adds a new configuration option `skip_newline`. If set, Filebeat does not add a newline when two events are concatenated. Closes #18038 (cherry picked from commit e3f51ab)
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
elastic#18352) ## What does this PR do? This PR adds a new mode for the multiline reader of Libbeat (exposed in Filebeat). The new mode lets users to aggregate the configured number of lines into a single event. Example configuration to aggregate 5 lines: ```yaml muliline.type: count multiline.count_lines: 5 ``` This PR also adds a new configuration option `skip_newline`. If set, Filebeat does not add a newline when two events are concatenated. Closes elastic#18038
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a new mode for the multiline reader of Libbeat (exposed in Filebeat). The new mode lets users to aggregate the configured number of lines into a single event.
Example configuration to aggregate 5 lines:
This PR also adds a new configuration option
skip_newline. If set, Filebeat does not add a newline when two events are concatenated.Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues
Closes #18038