Filebeat field "count" is created directly under the root of an event

[kurromat]

When forwarding files with filebeat (1.0.1) to logstash, I see a new field called "count" that seems to always contain the value 1.

It would be great if this field could be moved into the beat section of an event, since it right now overlays a field that should be created by logstash.

Remark: I have seen this field on both linux and windows as well as on filebeat and topbeat.

This is what is coming out of logstash:

{
"message" => "2016-01-19 17:00:05,473 INFO loggingtext",
"@Version" => "1",
"@timestamp" => "2016-01-19T16:00:05.473Z",
"beat" => {
"hostname" => "somehostname",
"name" => "somehostname"
},
"count" => 1,
"fields" => {
"type" => "MyType"
},
"input_type" => "log",
"offset" => 28754,
"source" => [
[0] "MyFile"
],
"type" => "log",
"host" => "MyHost",
"MyAttribute" => "Content",
"tags" => [
[0] "MyTag"
]
}

[ruflin]

@kurromat Sorry for the late reply. I actually think this field is not necessary in filebeat and we can remove it.

@andrewkroh It is set here. Is there a reason we need it? https://github.com/elastic/beats/blob/master/filebeat/input/file.go#L67

[andrewkroh]

All Beats have a count field. This is guaranteed by libbeat. My assumption is that this was added for Packetbeat to for the purposes of sampling. It doesn't seem to serve a purpose for most Beats.

If we use count only for Packetbeat I would retest all dashboards to make sure nothing is broken. Using a "sum of count" aggregation is common in the dashboards from what I recall.

[monicasarbu]

Andrew is right. The field count is exported by all the Beats and it is not used by any Beat. If it overlays with another field, I would say to just remove it for now and add it back when we start implementing sampling.
cc-ed @tsg

[monicasarbu]

It's value is always 1.

[tsg]

+1 to removing it, but we have to be careful that it might be referenced by the dashboards.

[ruflin]

For the moment I would only remove it for filebeat (which doesn't have dashboards).

[ruflin]

#1166 removes it for filebeat

[z0rc]

Can we reopen this one? I'm using filebeat 1.3.0 and the count field is here. Same config with filebeat shipping to logstash via beats input. And it's always 1.

[andrewkroh]

This was changed only in 5.x because it is non-backwards compatible change to the event schema. So you won't see it in 1.3, but you can try it in the 5.0.0-alpha6 release available on the website.