Beats metadata changes in 7.x - Hostname Unknown

[philomory]

In Filebeat 7.x (and in Beats 7.x in general), the path to certain metadata has changed. In particular, what was formerly under the beats key is now under the agent key, and beats.hostname has become agent.hostname. Because the Beats2 codec looks for beats.hostname to determine the sender of the message, all messages sent by Filebeat 7.x (and presumably messages sent by other Beats such as AuditBeat) now have source: unknown.

Expected Behavior

The Beats2 input should correctly determine the hostname of the sender regardless of the version of the Beat the message was sent by.

Current Behavior

The Beats2 input can only determine the hostname of the sender for Beats versions 6.x or lower, and fails to determine the hostname from messages sent by Beats versions 7.x or higher.

Possible Solution

Check the @metadata.version field and switch which parent field to look at on line 91 based on the results.

Alternatively, simply check for both agent.hostname and beats.hostname, picking whichever one actually exists.

Steps to Reproduce (for bugs)

1. Create a Beats2 Input
2. Install Filebeat 7.x on a system, with Graylog's Beats2 input configured as an output.
3. Produce some messages which will be ingested by Filebeat and sent to Graylog.
4. Observe that Graylog lists the source as "unknown"
5. Optionally, check Filebeat's debug log and see that the hostname key is now a child of the agent key rather than the beat key (which no longer exists at all).

Context

We've updated to Filebeat 7.x in some environments and discovered that message source is no longer recorded correctly. Of course, the required information is still present in the message, so we can work around this with pipeline processing, but this is something that should work out of the box.

Your Environment

• Graylog Version: 3.1.2
• Elasticsearch Version: 6.5.2
• MongoDB Version: 3.0.15
• Operating System: CentOS Linux release 7.5.1804 (Core)

[gimmic]

I would only consider everything 6.x in the elastic stack compatible with graylog.

[vinzol]

I also encountered such a problem, I hope you can solve it. Thank you very much!

[redhatromero]

Winlogbeat 7.7.1 issue still exists

[mpfz0r]

@redhatromero hmm, I tested this with 7.6.2. Which Graylog version are you using?

[redhatromero]

Graylog 3.2.4

[mpfz0r]

Graylog 3.2.4

This has been fixed in 3.2.5. you need to upgrade

[redhatromero]

ok... I will do that. thanks for the feedback.

[redhatromero]

once upgraded to the latest, currently 3.2.6, im no longer seeing ms windows servers appear in the ui as "source:unknown".

this does not seem to affect any logs captured prior to the upgrade but the current and going forward look as though they will appear as expected.

thanks @mpfz0r