Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep original messages in case of Filebeat modules #8448

Closed
wants to merge 7 commits into from
Closed

Keep original messages in case of Filebeat modules #8448

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9f2ee77
Keep original messages in case of Filebeat modules
kvch Sep 26, 2018
144c228
add changelog entry
kvch Sep 26, 2018
016484c
mv changelog entry
kvch Sep 28, 2018
df4304b
rename field to log.original
kvch Sep 28, 2018
d89be27
update config references
kvch Sep 28, 2018
32eb901
rm unused import
kvch Sep 28, 2018
da242a3
followup in tests
kvch Sep 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]

*Filebeat*

- Keep original messages in case of Filebeat modules. {pull}8448[8448]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not put it under breaking change as it's and addition but we should definitively have a note in the migration guide about the additional storage use.


*Heartbeat*

*Metricbeat*
Expand Down
6 changes: 6 additions & 0 deletions filebeat/_meta/fields.common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,12 @@
description: >
This field contains the flags of the event.

- name: log.message
type: keyword
description: >
The unprocessed original log message. This can be used for reprocessing logs.
index: false

- name: event.created
type: date
description: >
Expand Down
28 changes: 20 additions & 8 deletions filebeat/channel/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package channel
import (
"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
"github.com/elastic/beats/libbeat/processors"
)

Expand All @@ -43,6 +44,9 @@ type clientEventer struct {
// inputOutletConfig defines common input settings
// for the publisher pipeline.
type inputOutletConfig struct {
// KeepOriginalMsg determines if the original message needs to be kept for a module.
KeepOriginalMsg bool `config:"keep_original_message"`

// event processing
common.EventMetadata `config:",inline"` // Fields and tags to add to events.
Processors processors.PluginConfig `config:"processors"`
Expand All @@ -59,6 +63,10 @@ type inputOutletConfig struct {

}

var defaultConfig = inputOutletConfig{
KeepOriginalMsg: true,
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly believe this should be a opt-in feature.

// NewOutletFactory creates a new outlet factory for
// connecting an input to the publisher pipeline.
func NewOutletFactory(
Expand All @@ -82,7 +90,7 @@ func NewOutletFactory(
// This guarantees ordering between events as required by the registrar for
// file.State updates
func (f *OutletFactory) Create(p beat.Pipeline, cfg *common.Config, dynFields *common.MapStrPointer) (Outleter, error) {
config := inputOutletConfig{}
config := defaultConfig
if err := cfg.Unpack(&config); err != nil {
return nil, err
}
Expand All @@ -101,13 +109,16 @@ func (f *OutletFactory) Create(p beat.Pipeline, cfg *common.Config, dynFields *c
meta := common.MapStr{}
setMeta(meta, "pipeline", config.Pipeline)

keepOriginal := false
fields := common.MapStr{}
setMeta(fields, "module", config.Module)
setMeta(fields, "name", config.Fileset)
if len(fields) > 0 {
fields = common.MapStr{
"fileset": fields,
}
keepOriginal = config.KeepOriginalMsg

}
if config.Type != "" {
fields["prospector"] = common.MapStr{
Expand All @@ -119,13 +130,14 @@ func (f *OutletFactory) Create(p beat.Pipeline, cfg *common.Config, dynFields *c
}

client, err := p.ConnectWith(beat.ClientConfig{
PublishMode: beat.GuaranteedSend,
EventMetadata: config.EventMetadata,
DynamicFields: dynFields,
Meta: meta,
Fields: fields,
Processor: processors,
Events: f.eventer,
PublishMode: beat.GuaranteedSend,
EventMetadata: config.EventMetadata,
DynamicFields: dynFields,
Meta: meta,
Fields: fields,
KeepOriginalMsg: keepOriginal,
Processor: processors,
Events: f.eventer,
})
if err != nil {
return nil, err
Expand Down
12 changes: 12 additions & 0 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3042,6 +3042,18 @@ Logging level.
This field contains the flags of the event.


--

*`log.message`*::
+
--
type: keyword

The unprocessed original log message. This can be used for reprocessing logs.


Field is not indexed.

--

*`event.created`*::
Expand Down
48 changes: 48 additions & 0 deletions filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Authorization logs
#auth:
Expand All @@ -42,6 +45,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#------------------------------- Apache2 Module ------------------------------
#- module: apache2
Expand All @@ -56,6 +62,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Error logs
#error:
Expand All @@ -68,6 +77,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#------------------------------- Auditd Module -------------------------------
#- module: auditd
Expand All @@ -81,6 +93,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not make statements about doubling the size without measuring it. I also think it's not needed to have this in the config file but we could add a note about it in the docs that the there is increase storage use.

#keep_original_message: true

#---------------------------- elasticsearch Module ---------------------------
- module: elasticsearch
Expand Down Expand Up @@ -142,6 +157,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Debug logs
#debug:
Expand All @@ -154,6 +172,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Startup logs
#startup:
Expand All @@ -166,6 +187,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#--------------------------------- IIS Module --------------------------------
#- module: iis
Expand All @@ -180,6 +204,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Error logs
#error:
Expand All @@ -192,6 +219,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#-------------------------------- Kafka Module -------------------------------
- module: kafka
Expand Down Expand Up @@ -250,6 +280,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#-------------------------------- MySQL Module -------------------------------
#- module: mysql
Expand All @@ -264,6 +297,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

# Slow logs
#slowlog:
Expand All @@ -276,6 +312,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#-------------------------------- Nginx Module -------------------------------
#- module: nginx
Expand All @@ -302,6 +341,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#------------------------------- Osquery Module ------------------------------
- module: osquery
Expand Down Expand Up @@ -330,6 +372,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true

#-------------------------------- Redis Module -------------------------------
#- module: redis
Expand Down Expand Up @@ -364,6 +409,9 @@ filebeat.modules:
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
#Keeps the original message, so the data can be processed again on Ingest Node
#It requires increased storage size, because the sizes of events are approximately doubled.
#keep_original_message: true


#=========================== Filebeat inputs =============================
Expand Down
Loading