Kafka beats implementation

[Nithin-Kasam]

Description

1.Pulls the system logs from Kafka topic by using Filebeats configuration.

Motivation and Context

1.This change eliminates reading the file from remote systems to get system logs, instead we get this logs from kafka topic improving the performance.

How Has This Been Tested?

1.Configured filebeats.yml file in remote system inside same network.
2.Installed kafka,zookeeper and created topic with name configured in filebeats.yml.
3.Started Consumer console and able to see logs from the topic.
4.Created Kafka beats input and verified logs in conusmer console and graylog.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[CLAassistant]

All committers have signed the CLA.

[ryan-carroll-graylog]

Thanks for getting this PR up @Nithin-Kasam! Code looks good to me, glad it looks like we could reuse so much of the existing code.

I plan to get set up for testing this in the next day or two, then I'll add my final review.

[ryan-carroll-graylog]

We should specify that this is a new Graylog Input:

Suggested change

	message="Display System logs from Kafka topic using file beats"
	message="Added new Beats Kafka input to ingest system logs from Kafka topic using Filebeat."

[ryan-carroll-graylog]

This all looks good and tests successfully for me, nice work @Nithin-Kasam!

Zack is planning to take a look at this soon too for one more set of eyes, then once he approves we should be good to merge this in.

[kingzacko1]

LGTM! Nice work on this.