Logstash docs and configs for Filebeat modules

[acchen97]

Filebeat modules (FBM) are brewing and will introduce a new, turnkey solution for popular industry logs with the Elastic Stack. Initially, this will be inclusive of Filebeat configs, ingest node pipeline configs, and Kibana dashboards. Logstash can be formally included in the future when there are config management and auto-deploy capabilities. We also have roadmap plans for more seamless migrations from ingest node to Logstash pipeline configs.

As a stop-gap to complete the full stack play at FBM initial release time, we will create documentation with module examples on how to integrate nicely with Logstash if users desire to do so.

Tasks:

• Manual translation of ingest node to LS configs per module
• New LS documentation page around ingest node migration guidance with a focus on the FBM use cases. This should be linked from the Beats side in the FBM documentation.

/cc @suyograo @jordansissel @tsg @brandonmensing @dedemorton

[acchen97]

With Filebeat modules basically done and going into 5.3, let's get this portion going on the LS side. The 4 modules that we have are apache, nginx, mysql, and system logs.

@tsg you had previously converting LS configs to ingest node configs for some of these. Are these apache and nginx examples the ones you used? Anything you could share around that would be great. On the LS side, I'd expect the configuration of the Beats input/ES output ports, hosts, etc. will likely have to be manual at this juncture. We also need to have a different Filebeat config for these too, since it'll be using the LS output instead of the ES output, right?

@suyograo who should help with the config translations?

@dedemorton are you available to help with this documentation? I suspect you may be helping out on the Beats side here as well. :)

[tsg]

My suggestion would be to provide a "shell" sample Logstash configuration and then snippets for each of the modules. For example, the shell config could be this:

input {
  beats {
    port => 5044
  }
}

# Copy per module snippets here.

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

And then a module snippet could be something like this (not sure this is actually valid, don't shoot me if it's wrong):

filter {
  if [fileset.module] == "nginx" {
    if [fileset.name] == "access" {
      grok {
       ...
      }
    }
    if [fileset.name] == "error" {
      grok {
      ...
      }
    }
  }
}

The users have to copy & paste the snippets they need to the the right place. I know we can do somehow better with config dir or similar, but I feel like for the beginning the above would be clearer.

WDYT?

Open questions:

• Is this something for the Beats docs or the Logstash docs?
• Should we put these sample configurations on a single page, or on each module documentation? The advantage of putting them in the Filebeat modules is that at some point we will be able to automatically test them.
• In Ingest Node, there's the option to specify multiple Grok patterns and the first one that matches is used. Does that work with the LS grok filter? Specifying multiple "match" keys seems to be supported but I'm not sure the semantics are the same.
• I think I saw somewhere that grok adds a tag on failure? This could be used to simulate the "on_failure" handler that IN has. Basically in Filebeat modules we use on_failure to set a key with the error and still index the document unparsed.

@acchen97 When writing the sample modules, I've started, for example, from the configuration files from https://github.com/elastic/examples but then modified them quite a bit to fit the Ingest Node model and the naming conventions of Filebeat. My suggestion would be to do the reverse translation, rather than using the original configurations.

[dedemorton]

@acchen97 It definitely makes sense for me to work on this.

@tsg I haven't put much thought into this, but my gut response is to put the Logstash config examples in the Beats doc because the examples are only relevant to Filebeat users. On the other hand, we're pretty sure that most Logstash users are using Filebeat for ingest. :-) At the very least, we should have a page in the LS docs that describes Filebeat modules and points off to the Filebeat docs for the detailed config. I really need to play around with the modules to figure out what makes the most sense from a user's perspective.

[acchen97]

@tsg thanks for the details. I was thinking a full LS config (input/filter/output) for each module.

Is this something for the Beats docs or the Logstash docs?

We should have a section in the LS docs that gives a quick summary of FBM and how LS plays into it. As FBM are a one click deploy experience, I think we can put the LS config examples in the LS docs, and link to them from the FBM docs. @dedemorton I'll let you make the call on this one.

Should we put these sample configurations on a single page, or on each module documentation? The advantage of putting them in the Filebeat modules is that at some point we will be able to automatically test them.

Per module makes sense to me.

In Ingest Node, there's the option to specify multiple Grok patterns and the first one that matches is used. Does that work with the LS grok filter? Specifying multiple "match" keys seems to be supported but I'm not sure the semantics are the same.

LS should be able to do something similar.

I think I saw somewhere that grok adds a tag on failure? This could be used to simulate the "on_failure" handler that IN has. Basically in Filebeat modules we use on_failure to set a key with the error and still index the document unparsed.

Yes, we have a _grokparsefailure tag and should be able to do something similar as well. When ingest node sees a grok failure, does it send the event to a different index?

[tsg]

Yes, we have a _grokparsefailure tag and should be able to do something similar as well. When ingest node sees a grok failure, does it send the event to a different index?

We currently use the same index in FBM. We just set an error key with the error.

[suyograo]

PR here: #6791

[brandonmensing]

When we get this issue resolved, we should update the note at the bottom of the FB modules overview: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html

[acchen97]

@dedemorton @suyog are with wrapped up with this one? From what I see, the PRs are merged.

[dedemorton]

More or less. We need to iterate on it a bit to fix the syslog example, but it's essentially done. I also need to check back with the Beats team to see if there are any changes for 5.4 that impact the steps.

[dedemorton]

Closing this because the initial work is done. We'll track upcoming work in #7188