Add raw log message to log handler #7207

ruflin · 2018-05-30T11:14:01Z

By default filebeat adds log.message to each event which contains the raw unprocessed message. This can be disabled by setting raw_message to false to save disk space.

The message in log.message contains a full multiline message, is encoded and the new line at the end is stripped.

ruflin · 2018-05-30T11:14:26Z

@tsg I'm not sure if we should enable this by default or only have it for example in all our modules enabled.

Current default for ignore_above is 1024. This is too short for some command line entries on Windows. This increases it to 2048. Closes elastic#8076

kvch · 2018-08-28T09:14:33Z

CHANGELOG.asciidoc

@@ -178,6 +178,8 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
 - Correctly join partial log lines when using `docker` input. {pull}6967[6967]
 - Add support for TLS with client authentication to the TCP input {pull}7056[7056]
 - Converted part of pipeline from treafik/access metricSet to dissect to improve efficeny. {pull}7209[7209]
+- Add support for TLS with client authentication to the TCP input. {pull}7056[7056]


This entry does not belong here.

kvch · 2018-08-28T09:15:34Z

CHANGELOG.asciidoc

@@ -178,6 +178,8 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
 - Correctly join partial log lines when using `docker` input. {pull}6967[6967]
 - Add support for TLS with client authentication to the TCP input {pull}7056[7056]
 - Converted part of pipeline from treafik/access metricSet to dissect to improve efficeny. {pull}7209[7209]
+- Add support for TLS with client authentication to the TCP input. {pull}7056[7056]
+- Add log.message to each log event. {pull}[]


Please, add the number of the PR.

kvch

We agreed that this feature is going to be disabled by default, because it has a big impact on users. If it is enabled silently on update for users, their whole installation could break. The size of the events are doubled, so they run out of storage. At this point this must be a conscious decision of users to enable it. Thus, we ensure they can prepare for/acknowledge the increased storage requirements.
If we decide later to enable it by default, we can do it in 7.x.

kvch · 2018-08-28T09:37:50Z

I've just looked at the issue which references it. I still believe that on input level, raw should not be added to the event. But I am fine with adding this to the modules. I am keeping my review as request changes, because currently it enables keeping the raw message for log input, not just for modules.

kvch · 2018-08-28T09:47:51Z

filebeat/harvester/reader/limit.go

@@ -18,5 +18,9 @@ func (p *Limit) Next() (Message, error) {
 	if len(message.Content) > p.maxBytes {
 		message.Content = message.Content[:p.maxBytes]
 	}
+
+	if len(message.Raw) > p.maxBytes {
+		message.Raw = message.Raw[:p.maxBytes]


I am on the fence with this. Raw message should not be truncated, because it is a form of processing. But this could protect users from injecting too long messages to their pipelines.

I would go with truncating the raw message.

I would actually go with your first proposal and not truncate it. At the same time I would not index log.original.

Can you elaborate on what you mean with too long message to the pipeline?

I mean such a big message which could disrupt the output of Filebeat.

By default filebeat adds log.message to each event which contains the raw unprocessed message. This can be disabled by setting `raw_message` to false to save disk space. The message in `log.message` contains a full multiline message, is encoded and the new line at the end is stripped.

ruflin

It would be interesting to see what the actual impact on enabling it would mean on the storage level. My hope is that it has a smaller impact as the same data exists twice it "could" compress well.

@kvch @ph This change currently only applies to the log input. Should we to something similar for the other inputs?

ruflin · 2018-08-29T09:53:57Z

filebeat/harvester/reader/limit.go

@@ -18,5 +18,9 @@ func (p *Limit) Next() (Message, error) {
 	if len(message.Content) > p.maxBytes {
 		message.Content = message.Content[:p.maxBytes]
 	}
+
+	if len(message.Raw) > p.maxBytes {
+		message.Raw = message.Raw[:p.maxBytes]


I would actually go with your first proposal and not truncate it. At the same time I would not index log.original.

Can you elaborate on what you mean with too long message to the pipeline?

kvch · 2018-08-30T14:05:16Z

I think we should. In case of log input reprocessing can be achieved by rereading the data (if it's not yet deleted). But for other inputs it might not be straightforward to replay the incoming messages. So it might be more valuable for other inputs than for log.

ruflin · 2018-09-27T09:32:32Z

Closing in favor of #8448

ruflin added in progress Pull request is currently in progress. discuss Issue needs further discussion. review Filebeat Filebeat labels May 30, 2018

ruflin mentioned this pull request May 30, 2018

Add log.message elastic/ecs#3

Merged

ruflin force-pushed the raw-message branch 2 times, most recently from e5e67b8 to 9eedc54 Compare June 4, 2018 07:03

Increase ignore_above to 2048 for system.process.cmdline

b44a5f7

Current default for ignore_above is 1024. This is too short for some command line entries on Windows. This increases it to 2048. Closes elastic#8076

ruflin mentioned this pull request Aug 27, 2018

Filebeat modules: keep raw message #8083

Closed

urso requested a review from kvch August 27, 2018 15:01

kvch reviewed Aug 28, 2018

View reviewed changes

kvch suggested changes Aug 28, 2018

View reviewed changes

kvch reviewed Aug 28, 2018

View reviewed changes

ruflin added 6 commits August 28, 2018 13:40

Merge branch 'master' of github.com:ruflin/beats

4df9ed2

rebase fields

fe3d3c3

fix docker tests, problem was that log and log.message conflict

306a25e

rebase and fix changelog

2790c80

rebase and rename to original

ba24d6a

ruflin force-pushed the raw-message branch from 9eedc54 to ba24d6a Compare August 29, 2018 10:07

ruflin commented Aug 29, 2018

View reviewed changes

fix harvester and changelog for original

0102a0b

kvch mentioned this pull request Sep 26, 2018

Keep original messages in case of Filebeat modules #8448

Closed

ruflin closed this Sep 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add raw log message to log handler #7207

Add raw log message to log handler #7207

ruflin commented May 30, 2018

ruflin commented May 30, 2018

kvch Aug 28, 2018

kvch Aug 28, 2018

kvch left a comment

kvch commented Aug 28, 2018 •

edited

Loading

kvch Aug 28, 2018

kvch Aug 28, 2018

ruflin Aug 29, 2018

kvch Aug 30, 2018

ruflin left a comment

ruflin Aug 29, 2018

kvch commented Aug 30, 2018

ruflin commented Sep 27, 2018

Add raw log message to log handler #7207

Add raw log message to log handler #7207

Conversation

ruflin commented May 30, 2018

ruflin commented May 30, 2018

kvch Aug 28, 2018

Choose a reason for hiding this comment

kvch Aug 28, 2018

Choose a reason for hiding this comment

kvch left a comment

Choose a reason for hiding this comment

kvch commented Aug 28, 2018 • edited Loading

kvch Aug 28, 2018

Choose a reason for hiding this comment

kvch Aug 28, 2018

Choose a reason for hiding this comment

ruflin Aug 29, 2018

Choose a reason for hiding this comment

kvch Aug 30, 2018

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ruflin Aug 29, 2018

Choose a reason for hiding this comment

kvch commented Aug 30, 2018

ruflin commented Sep 27, 2018

kvch commented Aug 28, 2018 •

edited

Loading