Filebeat module for zookeeper logs

[matschaffer]

Describe the enhancement:

We have a lot of great modules in https://github.com/elastic/beats/tree/master/x-pack/filebeat/module but nothing for https://zookeeper.apache.org/

It'd be great to have a module with pre-defined beats & ingest pipeline configs to get those logs into ECS format.

Describe a specific use case for the enhancement or feature:

We run zookeeper as part of backend infrastructure for https://cloud.elastic.co/ and would like to ingest structured logs.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[legoguy1000]

can u post sample logs. I think this would be very interesting. I see the audit logs here, https://zookeeper.apache.org/doc/r3.7.0/zookeeperAuditLogs.html. Are there other logs?

[legoguy1000]

I've created a draft PR to add a module for ZooKeeper. I currently only have the audit log fileset. If there are other log types, please let me know and provide sample data.

[matschaffer]

nice. Thanks @legoguy1000! Here's a sample zookeeper.log from a dev env.

zookeeper.log.zip

Alternatively, we could probably change the logging output format rather than grokking (zookeeper just uses log4j).

Not sure what sort of precedent filebeat modules have in that space though. Maybe better to have modules just assume default output configuration.

[legoguy1000]

Those logs are disgusting. Working on a basic grok pattern now. Will probably need help to parse the actual messages. Using https://stackoverflow.com/questions/45569100/what-is-the-format-of-zookeeper-logs and to parse the format as best I can https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html

[legoguy1000]

I've updated the PR with the zookeeper.log fileset. Can you let me know if/how to parse the logs more? Or if you think its good, I can make it not draft.