From 1e5f1d2492ac3f0435ffa279081893f44a20eb31 Mon Sep 17 00:00:00 2001 From: DeDe Morton Date: Tue, 5 Sep 2017 12:37:10 -0700 Subject: [PATCH] [Docs] Document how to use modules.d directory (#4973) * [Docs] Document how to use modules.d directory * Add changes from review --- filebeat/docs/configuring-howto.asciidoc | 3 + .../docs/filebeat-modules-options.asciidoc | 194 +++++++++++++++--- filebeat/docs/filebeat-options.asciidoc | 6 + filebeat/docs/getting-started.asciidoc | 2 +- .../docs/modules-getting-started.asciidoc | 39 ++-- filebeat/docs/modules-overview.asciidoc | 37 ++-- libbeat/docs/command-reference.asciidoc | 8 +- 7 files changed, 223 insertions(+), 66 deletions(-) diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc index 745b8ab45f5..3a2d4d2f4a3 100644 --- a/filebeat/docs/configuring-howto.asciidoc +++ b/filebeat/docs/configuring-howto.asciidoc @@ -19,6 +19,7 @@ _Beats Platform Reference_ for more about the structure of the config file. The following topics describe how to configure Filebeat: +* <> * <> * <> * <> @@ -39,6 +40,8 @@ The following topics describe how to configure Filebeat: -- +include::./filebeat-modules-options.asciidoc[] + include::./filebeat-options.asciidoc[] include::./multiple-prospectors.asciidoc[] diff --git a/filebeat/docs/filebeat-modules-options.asciidoc b/filebeat/docs/filebeat-modules-options.asciidoc index 865abec98bc..a680464a574 100644 --- a/filebeat/docs/filebeat-modules-options.asciidoc +++ b/filebeat/docs/filebeat-modules-options.asciidoc @@ -1,28 +1,105 @@ [[configuration-filebeat-modules]] == Specify which modules to run -//REVIEWERS: I created this topic because I think it might be confusing to have no mention of this modules config section in the docs. However, I think that adding the info might also be confusing becasue users might try to configure othe things (like prospectors). Note that this topic uses the old command syntax, but would be updated to reflect the new. +NOTE: Using Filebeat modules is optional. You may decide to +<> if you are using +a log file type that isn't supported, or you want to use a different setup. Filebeat <> provide a quick way for you to get started processing common log formats. They contain default configurations, -Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help -you implement and deploy a log monitoring solution. +Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help you +implement and deploy a log monitoring solution. -Using modules is optional. You may decide to configure Filebeat manaully if -you are using a log file type that isn't supported or you want to use a -different setup. +Filebeat provides a few different ways to enable modules. You can: -To enable specific modules, you can add entries to the `filebeat.modules` list -in the +{beatname_lc}.yml+ config file. Each entry in the list begins with a dash -(-) and is followed by settings for that module. +* <> +* <> +* <> -Filebeat also provides command-line options for enabling and disabling modules. -See <> for more about running modules. +When you enable modules, you can also +<> to change the default +behavior of the modules, and you can specify +<> to override prospector settings. -The following example shows a configuration that runs the `nginx`,`mysql`, and +Before running Filebeat with modules enabled, make sure you also set up the +environment to use Kibana dashboards. See <> for +more information. + +[float] +[[enable-modules-d-configs]] +=== Enable module configs in the `modules.d` directory + +The `modules.d` directory contains default configurations for all the modules +available in Filebeat. You can enable or disable specific module configurations +under `modules.d` by running the +<> commands. + +For example, to enable the `apache2` and `mysql` configs in the `modules.d` +directory, you use: + +[source,shell] +---- +./filebeat modules enable apache2 mysql +---- + +Then when you run Filebeat, it loads the corresponding module configurations +specified in the `modules.d` directory (for example, `modules.d/apache2.yml` and +`modules.d/mysql.yml`). + +To see a list of enabled and disabled modules, run: + +[source,shell] +---- +./filebeat modules list +---- + +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. + +[float] +[[enable-modules-cli]] +=== Enable modules when you run Filebeat + +To enable specific <> when you run Filebeat at the +command line, you can use the `--modules` flag. This approach works well when +you're getting started and want to specify different modules and settings each +time you run Filebeat. Any modules specified at the command line will be loaded +along with any modules that are enabled in the configuration file or `modules.d` +directory. If there's a conflict, the configuration specified at the command +line is used. + +The following example shows how to enable and run the `nginx`,`mysql`, and `system` modules. -//QUESTION: Are we doing something like the modules.d layout for Filebeat? +[source,shell] +---- +./filebeat -e --modules nginx,mysql,system +---- + +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. + +[float] +[[enable-modules-config-file]] +=== Enable module configs in the +{beatname_lc}.yml+ file + +Enabling <> directly in the config file is a practical +approach if you have upgraded from a previous version of {beatname_uc} and don't +want to move your module configs to the `modules.d` directory. You can continue +to configure modules in the +{beatname_lc}.yml+ file, but you won't be able to +use the `modules` command to enable and disable configurations because the +command requires the `modules.d` layout. + +To enable specific modules in the +{beatname_lc}.yml+ config file, you can add +entries to the +{beatname_lc}.modules+ list. Each entry in the list begins with +a dash (-) and is followed by settings for that module. + +The following example shows a configuration that runs the `nginx`,`mysql`, and +`system` modules. [source,yaml] ---- @@ -32,34 +109,93 @@ filebeat.modules: - module: system ---- -To enable this same configuration from the command line, you use: +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. -[source,shell] ----- -./filebeat -e -modules=nginx,mysql,system ----- +[[specify-variable-settings]] +=== Specify variable settings -The configuration in the example assumes that the logs you’re harvesting are -in the location expected for your OS and that the default behavior of Filebeat -is appropriate for your environment. Each module provides variables that you -can set to fine tune the behavior of Filebeat, including the location -where it looks for log files. For example, the following configuration sets -the path for `nginx` log files: +Each module and fileset has variables that you can set to change the default +behavior of the module, including the paths where the module looks for log +files. For example, the `var.paths` setting in the following example sets the +path for `nginx` access log files: [source,yaml] ---- -filebeat.modules: - module: nginx access: var.paths: ["/var/log/nginx/access.log*"] ---- -To set the same configuration from the command line, you use: +To set the path for Nginx access log files at the command line, you use +the `-M` flag. For example: [source,shell] ---- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---- -See the <> you are using for -more information. +When you set variables at the command line, the variable name needs to include +the module and fileset name. You can specify multiple overrides. Each override +must start with `-M`. + +Here you see how to use the `-M` flag along with the `--modules` flag. This +example shows how to set the paths to the access and error logs: + +[source,shell] +---- +./filebeat --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" -M "nginx.error.var.paths=[/var/log/nginx/error.log*]" +---- + +For information about specific variables that you can set for each fileset, +see the <>. + +[[advanced-settings]] +=== Advanced settings + +Behind the scenes, each module starts a Filebeat prospector. Advanced users +can add or override any prospector settings. For example, you can set +<> to `true` in the module configuration: + +[source,yaml] +---------------------------------------------------------------------- +- module: nginx + access: + prospector: + close_eof: true +---------------------------------------------------------------------- + +Or at the command line like this: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "nginx.access.prospector.close_eof=true" +---------------------------------------------------------------------- + + +Here you see how to use the `-M` flag along with the `--modules` flag: + +[source,shell] +---------------------------------------------------------------------- +./filebeat --modules nginx -M "nginx.access.prospector.close_eof=true" +---------------------------------------------------------------------- + + +You can use wildcards to change variables or settings for multiple +modules/filesets at once. For example, the following command enables +`close_eof` for all the filesets in the `nginx` module: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "nginx.*.prospector.close_eof=true" +---------------------------------------------------------------------- + +The following command enables `close_eof` for all prospectors created by any of +the modules: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "*.*.prospector.close_eof=true" +---------------------------------------------------------------------- diff --git a/filebeat/docs/filebeat-options.asciidoc b/filebeat/docs/filebeat-options.asciidoc index 8458096de10..9a34bbb8baf 100644 --- a/filebeat/docs/filebeat-options.asciidoc +++ b/filebeat/docs/filebeat-options.asciidoc @@ -1,6 +1,12 @@ [[configuration-filebeat-options]] == Set up prospectors +TIP: <> provide the fastest getting +started experience for common log formats. See <> +to learn how to get started with modules. Also see +<> for information about enabling and +configuring modules. + Filebeat uses prospectors to locate and process files. To configure Filebeat, you specify a list of prospectors in the `filebeat.prospectors` section of the +{beatname_lc}.yml+ config file. diff --git a/filebeat/docs/getting-started.asciidoc b/filebeat/docs/getting-started.asciidoc index e09a06a545f..9406d724144 100644 --- a/filebeat/docs/getting-started.asciidoc +++ b/filebeat/docs/getting-started.asciidoc @@ -264,7 +264,7 @@ sudo chown root filebeat.yml <1> sudo ./filebeat -e -c filebeat.yml -d "publish" ---------------------------------------------------------------------- <1> You'll be running Filebeat as root, so you need to change ownership -of the configuration file, or run Filebeat with `-strict.perms=false` +of the configuration file, or run Filebeat with `--strict.perms=false` specified. See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] in the _Beats Platform Reference_. diff --git a/filebeat/docs/modules-getting-started.asciidoc b/filebeat/docs/modules-getting-started.asciidoc index 6191105c6df..cb387a57c1d 100644 --- a/filebeat/docs/modules-getting-started.asciidoc +++ b/filebeat/docs/modules-getting-started.asciidoc @@ -40,6 +40,9 @@ sudo bin/elasticsearch-plugin install ingest-user-agent ---------------------------------------------------------------------- + You need to restart Elasticsearch after running these commands. ++ +If you are using an https://cloud.elastic.co/[Elastic Cloud] instance, you can +enable the two plugins from the configuration page. * Verify that Elasticsearch and Kibana are running and that Elasticsearch is ready to receive data from Filebeat. @@ -58,17 +61,17 @@ the sample dashboards for visualizing the data in Kibana. For example: ./filebeat setup -e ---------------------------------------------------------------------- + -The value that you pass with the `-modules` flag is a comma-separated list of +The value that you pass with the `--modules` flag is a comma-separated list of modules that you want to set up. The `-e` flag is optional and sends output to standard error instead of syslog. -. Start Filebeat and use the `-modules` flag to specify the list of modules +. Start Filebeat and use the `--modules` flag to specify the list of modules you want to run. The following example starts Filebeat with the `system` module enabled (it's assumed that you've already loaded the sample dashboards): + [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=system +./filebeat -e --modules system ---------------------------------------------------------------------- + This command takes care of configuring Filebeat and loading the ingest node @@ -79,9 +82,19 @@ example: + [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=system,nginx,mysql +./filebeat -e --modules system,nginx,mysql ---------------------------------------------------------------------- - ++ +When you start Filebeat, you should see messages indicating that Filebeat +has started harvesters for all enabled modules. For example: ++ +[source,shell] +---------------------------------------------------------------------- +2017/08/16 23:39:15.414375 harvester.go:206: INFO Harvester started for file: /var/log/displaypolicyd.stdout.log +---------------------------------------------------------------------- ++ +If you don't see this message for each log file that needs to be read, +see <> to find out how to set the path the files. NOTE: Depending on how you've installed Filebeat, you might see errors related to file ownership or permissions when you try to run Filebeat modules. @@ -89,12 +102,10 @@ See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions in the _Beats Platform Reference_ if you encounter errors related to file ownership or permissions. -//include::system-module-note.asciidoc[] - - -TIP: In a production environment, you'll probably want to use a configuration -file, rather than command-line flags, to specify which modules to run. See the -detailed documentation for more about configuring and running modules. +This getting started guide uses the `--modules` flag to enable modules +at the command line when you run Filebeat. In a production environment, you'll +probably want to use the configs in the `modules.d` directory instead. See +<> for more information. [[setting-variables]] ==== Set the path variable @@ -108,11 +119,11 @@ logs: [source,shell] ---- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---- -See the <> for more information about setting variables and -advanced options. +See <> for more information about setting +variables and advanced options. [[passing-credentials-modules]] ==== Pass credentials diff --git a/filebeat/docs/modules-overview.asciidoc b/filebeat/docs/modules-overview.asciidoc index 0f9c363e58f..f182f48af2c 100644 --- a/filebeat/docs/modules-overview.asciidoc +++ b/filebeat/docs/modules-overview.asciidoc @@ -34,9 +34,20 @@ Node. Filebeat modules require Elasticsearch 5.2 or later. +[float] +=== Get started + +To learn how to configure and run Filebeat modules: + +* Get started by reading <>. +* Try out the extended <> for a more in-depth introduction. +* Learn about the different ways to enable modules in <>. + [[modules-tutorial]] === Tutorial +//TODO: Replace this content with a more in-depth tutorial or remove it. + This tutorial assumes you have Elasticsearch and Kibana installed and accessible from Filebeat (see the <> section). It also assumes that the Ingest Node GeoIP and User Agent plugins are @@ -63,13 +74,13 @@ You can start Filebeat with the following command: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -setup +./filebeat -e --modules nginx -setup ---------------------------------------------------------------------- The `-e` flag tells Filebeat to output its logs to standard error, instead of syslog. -The `-modules=nginx` flag loads the Nginx module. +The `--modules nginx` flag loads the Nginx module. The `-setup` flag tells Filebeat to load the associated sample Kibana dashboards. This setup phase, in which the dashboards are loaded, doesn't have @@ -86,10 +97,9 @@ You can also start multiple modules at once: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx,mysql,system +./filebeat -e --modules nginx,mysql,system ---------------------------------------------------------------------- -include::system-module-note.asciidoc[] While enabling the modules from the CLI file is handy for getting started and for testing, you will probably want to use the configuration file for the @@ -123,7 +133,7 @@ files are in a custom location: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---------------------------------------------------------------------- Or via the configuration file: @@ -136,17 +146,6 @@ filebeat.modules: var.paths: ["/var/log/nginx/access.log*"] ---------------------------------------------------------------------- -The Nginx `access` fileset also has a `pipeline` variable which allows -selecting which of the available Ingest Node pipelines is used for parsing. At -the moment, two such pipelines are available, one that requires the two ingest -plugins (`ingest-geoip` and `ingest-user-agent`) and one that doesn't. If you -cannot install the plugins, you can use the following: - - -[source,shell] ----------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.var.pipeline=no_plugins" ----------------------------------------------------------------------- ==== Advanced settings @@ -169,7 +168,7 @@ Or like this: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true" +./filebeat -e --modules nginx -M "nginx.access.prospector.close_eof=true" ---------------------------------------------------------------------- From the CLI, it's possible to change variables or settings for multiple @@ -178,7 +177,7 @@ modules/fileset at once. For example, the following works and will enable [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true" +./filebeat -e --modules nginx -M "nginx.*.prospector.close_eof=true" ---------------------------------------------------------------------- The following also works and will enable `close_eof` for all prospectors @@ -186,5 +185,5 @@ created by any of the modules: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true" +./filebeat -e --modules nginx,mysql -M "*.*.prospector.close_eof=true" ---------------------------------------------------------------------- diff --git a/libbeat/docs/command-reference.asciidoc b/libbeat/docs/command-reference.asciidoc index 3fe3759c1e6..a13c612ef42 100644 --- a/libbeat/docs/command-reference.asciidoc +++ b/libbeat/docs/command-reference.asciidoc @@ -159,9 +159,11 @@ ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")] ==== `modules` command {modules-command-short-desc}. You can use this command to enable and disable -specific modules. The changes you make with this command are persisted and -used for subsequent runs of {beatname_uc}. To see which modules are enabled -and disabled, run the `list` subcommand. +specific module configurations defined in the `modules.d` directory. The +changes you make with this command are persisted and used for subsequent +runs of {beatname_uc}. + +To see which modules are enabled and disabled, run the `list` subcommand. *SYNOPSIS*