TCP/TLS and/or syslog output for beats

[candlerb]

Describe the enhancement:

Add TCP/TLS output to beats.

1. Plain TCP output delimited by \n, configurable with or without SSL (directly compatible with beats TCP input feature)
2. Optionally, each message is instead prefixed by frame length (RFC5425 section 4.3)
3. Optionally, full syslog format (RFC5424) which means prefixing the payload with priority, version, timestamp, hostname etc.

(Taken to extreme, could also decompose JSON into RFC5424 structured-data. I don't think that's necessary)

Describe a specific use case for the enhancement or feature:

syslog is the de-facto protocol for centralising logs. Using TCP and TLS it is also reliable and secure.

Having syslog output would allow direct integration into existing pipelines which use syslog as the log aggregation protocol. This makes a wider range of tools available, and beats much more attractive as general-purpose collectors.

Having plain TCP output would allow easy integration with custom code. It would also allow edge beats to talk to a central filebeat instance (with TCP input). This would give a simple and cheap way to aggregate logs, without a heavyweight logstash server.

TCP/TLS would be an alternative to beats/TLS as a secure transport, and easier to integrate with.

Beats currently support only a limited set of outputs and these all have issues in a heterogenous environment:

• Logstash (lumberjack) output has no corresponding beats input (beats (lumberjack) input plugin for filebeat #10890). This means that you are forced to run logstash to receive the logs, which is considered by some to be the most troublesome part of the elastic stack.
• Redis output is insecure across a network for multiple reasons. You don't want your redis server to be opened up to your entire infrastructure.
• Kafka output is limited to those running a Kafka cluster, and who are happy to open the cluster to direct communication, and configure direct Kafka output from all edge nodes
• Elasticsearch output bypasses your log processing pipeline, and also requires exposing ES read/write to your entire infrastructure.

Being able to deliver to a traditional syslog stack opens up a whole bunch of options, which can still include delivery of logs into the rest of the elastic stack alongside other log processing.

Workarounds

There is a third-party beats forwarder, which can convert beats protocol to syslog, TCP or UDP. Having beats able to output TCP or syslog natively would remove the need for this moving part.

[rwaweber]

I would love to see this feature added as well! As I've hit a couple of the rubs you outline above myself.

Another benefit that this feature adds is being able to offer a loadbalance-able output to beats. This would make it possible to easily scale out a logstash or other system to handle many distinct clients shipping to a single endpoint.

I think we could distill this into two major outputs like you outline above:

• one being a generic, likely JSON-over-TCP w/ TLS options
• the other being a syslog output, compliant with RFC 5424

I think that implementing the former first will likely be simpler, as we won't need to spend as much time working out formatting bits. We could simply pass along the json structures that all other outputs use.

I think it can also lay the path for working out a cross-platform option for the latter given syslog implementations in go seem to primarily support unix-y systems without much support for windows:

• https://golang.org/pkg/log/syslog/#pkg-note-BUG
• https://github.com/hashicorp/go-syslog

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.