[Docs] Document how to use modules.d directory

filebeat/docs/configuring-howto.asciidoc

@@ -19,6 +19,7 @@ _Beats Platform Reference_ for more about the structure of the config file.
 
 The following topics describe how to configure Filebeat:
 
+* <<configuration-filebeat-modules>>
 * <<configuration-filebeat-options>>
 * <<multiline-examples>>
 * <<configuration-general-options>>
@@ -39,6 +40,8 @@ The following topics describe how to configure Filebeat:
 
 --
 
+include::./filebeat-modules-options.asciidoc[]
+
 include::./filebeat-options.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]

filebeat/docs/filebeat-modules-options.asciidoc

@@ -1,28 +1,107 @@
 [[configuration-filebeat-modules]]
 == Specify which modules to run
 
-//REVIEWERS: I created this topic because I think it might be confusing to have no mention of this modules config section in the docs. However, I think that adding the info might also be confusing becasue users might try to configure othe things (like prospectors). Note that this topic uses the old command syntax, but would be updated to reflect the new.
+NOTE: Using Filebeat modules is optional. You may decide to
+<<configuration-filebeat-options,set up prospectors manually>> if you are using
+a log file type that isn't supported, or you want to use a different setup.
 
 Filebeat <<filebeat-modules,modules>> provide a quick way for you to get started
 processing common log formats. They contain default configurations,
-Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help
-you implement and deploy a log monitoring solution.
+Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help you
+implement and deploy a log monitoring solution.
 
-Using modules is optional. You may decide to configure Filebeat manaully if
-you are using a log file type that isn't supported or you want to use a
-different setup.
+Filebeat provides a few different ways to enable modules. You can:
 
-To enable specific modules, you can add entries to the `filebeat.modules` list
-in the +{beatname_lc}.yml+ config file. Each entry in the list begins with a dash
-(-) and is followed by settings for that module.
+* <<enable-modules-d-configs>>
+* <<enable-modules-cli>>
+* <<enable-modules-config-file>>
 
-Filebeat also provides command-line options for enabling and disabling modules.
-See <<filebeat-modules-overview>> for more about running modules.
+When you enable modules, you can also
+<<override-variables,set variable overrides>> to change default settings, and
+you can specify <<advanced-settings,advanced settings>> to override prospector
+settings.
 
-The following example shows a configuration that runs the `nginx`,`mysql`, and
+Before running Filebeat with modules enabled, make sure you also set up the
+enviroment to use Kibana dashboards. See <<filebeat-modules-quickstart>> for
+more information.
+
+[float]
+[[enable-modules-d-configs]]
+=== Enable module configs in the `modules.d` directory
+
+The `modules.d` directory contains default configurations for all the modules
+available in Filebeat. You can enable or disable specific module configurations
+under `modules.d` by running the
+<<modules-command,`modules enable` or `modules disable`>> commands.
+
+For example, to enable the `apache2` and `mysql` configs in the `modules.d`
+directory, you use:
+
+[source,shell]
+----
+./filebeat modules enable apache2 mysql
+----
+
+Then when you run Filebeat, it loads the corresponding module configurations
+specified in the `modules.d` directory (for example, `modules.d/apache2.yml` and
+`modules.d/mysql.yml`).
+
+To see a list of enabled and disabled modules, run:
+
+[source,shell]
+----
+./filebeat modules list
+----
+
+The default module configurations assume that the logs you’re harvesting are
+in the location expected for your OS and that the default behavior of Filebeat
+is appropriate for your environment. To change the default settings, you need
+to specify variable overrides. See <<override-variables>>.
+
+[float]
+[[enable-modules-cli]]
+=== Enable modules when you run Filebeat
+
+To enable specific <<filebeat-modules,modules>> when you run Filebeat at the
+command line, you can use the `--modules` flag. This approach works well when
+you're getting started and want to specify different modules and settings each
+time you run Filebeat. Any modules specified at the command line will be loaded
+along with any modules that are enabled in the configuration file or `modules.d`
+directory. If there's a conflict, the configuration specified at the command
+line is used.
+
+//REVIEWERS: Is the above statement (about how Filebeat uses all module configs) true? I tested some combinations, but not all. Please verify.
+
+The following example shows how to enable and run the `nginx`,`mysql`, and
 `system` modules.
 
-//QUESTION: Are we doing something like the modules.d layout for Filebeat?
+[source,shell]
+----
+./filebeat -e --modules nginx,mysql,system
+----
+
+The default module configurations assume that the logs you’re harvesting are
+in the location expected for your OS and that the default behavior of Filebeat
+is appropriate for your environment. To change the default settings, you need
+to specify variable overrides at the command line. See <<override-variables>>.
+
+[float]
+[[enable-modules-config-file]]
+=== Enable module configs in the +{beatname_lc}.yml+ file
+
+Enabling <<filebeat-modules,modules>> directly in the config file is a practical
+approach if you have upgraded from a previous version of {beatname_uc} and don't
+want to move your module configs to the `modules.d` directory. You can continue
+to configure modules in the +{beatname_lc}.yml+ file, but you won't be able to
+use the `modules` command to enable and disable configurations because the
+command requires the `modules.d` layout.
+
+To enable specific modules in the +{beatname_lc}.yml+ config file, you can add
+entries to the `filebeat.modules` list. Each entry in the list begins with a
+dash (-) and is followed by settings for that module.
+
+The following example shows a configuration that runs the `nginx`,`mysql`, and
+`system` modules.
 
 [source,yaml]
 ----
@@ -32,34 +111,93 @@ filebeat.modules:
 - module: system
 ----
 
-To enable this same configuration from the command line, you use:
-
-[source,shell]
-----
-./filebeat -e -modules=nginx,mysql,system
-----
-
 The configuration in the example assumes that the logs you’re harvesting are
 in the location expected for your OS and that the default behavior of Filebeat
-is appropriate for your environment. Each module provides variables that you
-can set to fine tune the behavior of Filebeat, including the location
-where it looks for log files. For example, the following configuration sets
-the path for `nginx` log files:
+is appropriate for your environment. To change the default settings, you need
+to specify variable overrides. See <<override-variables>>.
+
+[[override-variables]]
+=== Override module settings
+
+Each module and fileset has a set of variables that you can set to override
+the default config, including the paths where the module looks for log files.
+For example, the `var.paths` setting in the following example sets the path
+for `nginx` access log files:
 
 [source,yaml]
 ----
-filebeat.modules:
 - module: nginx
   access:
     var.paths: ["/var/log/nginx/access.log*"]
 ----
 
-To set the same configuration from the command line, you use:
+To set the path for Nginx access log files at the command line, you use
+the `-M` flag. For example:
 
 [source,shell]
 ----
-./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
+./filebeat -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
 ----
 
-See the <<filebeat-modules,documentation for the modules>> you are using for
-more information.
+When you set variables at the command line, the variable name needs to include
+the module and fileset name. You can specify multiple overrides. Each override
+must start with `-M`.
+
+Here you see how to use the `-M` flag along with the `--modules` flag. This
+example shows how to set the paths to the access and error logs:
+
+[source,shell]
+----
+./filebeat --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" -M "nginx.error.var.paths=[/var/log/nginx/error.log*]"
+----
+
+For information about specific variables that you can override for each
+fileset, see the <<filebeat-modules,documentation for the modules>>.
+
+[[advanced-settings]]
+=== Advanced settings
+
+Behind the scenes, each module starts a Filebeat prospector. Advanced users
+can add or override any prospector settings. For example, you can set
+<<close-eof,close_eof>> to `true` in the module configuration:
+
+[source,yaml]
+----------------------------------------------------------------------
+- module: nginx
+  access:
+    prospector:
+      close_eof: true
+----------------------------------------------------------------------
+
+Or at the command line like this:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "nginx.access.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+
+Here you see how to use the `-M` flag along with the `--modules` flag:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat --modules nginx -M "nginx.access.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+
+You can use wildcards to change variables or settings for multiple
+modules/filesets at once. For example, the following command enables
+`close_eof` for all the filesets in the `nginx` module:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "nginx.*.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+The following command enables `close_eof` for all prospectors created by any of
+the modules:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "*.*.prospector.close_eof=true"
+----------------------------------------------------------------------

filebeat/docs/filebeat-options.asciidoc

@@ -1,6 +1,12 @@
 [[configuration-filebeat-options]]
 == Set up prospectors
 
+TIP: <<filebeat-modules-overview,Filebeat modules>> provide the fastest getting
+started experience for common log formats. See <<filebeat-modules-quickstart>>
+to learn how to get started with modules. Also see
+<<configuration-filebeat-modules>> for information about enabling and
+configuring modules.
+
 Filebeat uses prospectors to locate and process files. To configure Filebeat,
 you specify a list of prospectors in the `filebeat.prospectors` section of the
 +{beatname_lc}.yml+ config file.

filebeat/docs/getting-started.asciidoc

@@ -264,7 +264,7 @@ sudo chown root filebeat.yml <1>
 sudo ./filebeat -e -c filebeat.yml -d "publish"
 ----------------------------------------------------------------------
 <1> You'll be running Filebeat as root, so you need to change ownership
-of the configuration file, or run Filebeat with `-strict.perms=false`
+of the configuration file, or run Filebeat with `--strict.perms=false`
 specified. See
 {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
 in the _Beats Platform Reference_.

filebeat/docs/modules-getting-started.asciidoc

@@ -40,6 +40,9 @@ sudo bin/elasticsearch-plugin install ingest-user-agent
 ----------------------------------------------------------------------
 +
 You need to restart Elasticsearch after running these commands.
++
+If you are using an https://cloud.elastic.co/[Elastic Cloud] instance, you can
+enable the two plugins from the configuration page.
 
 * Verify that Elasticsearch and Kibana are running and that Elasticsearch is
 ready to receive data from Filebeat.
@@ -81,20 +84,28 @@ example:
 ----------------------------------------------------------------------
 ./filebeat -e -modules=system,nginx,mysql
 ----------------------------------------------------------------------
-
++
+When you start Filebeat, you should see messages indicating that Filebeat
+has started harvesters for all enabled modules. For example:
++
+[source,shell]
+----------------------------------------------------------------------
+2017/08/16 23:39:15.414375 harvester.go:206: INFO Harvester started for file: /var/log/displaypolicyd.stdout.log
+----------------------------------------------------------------------
++
+If you don't see this message for each log file that needs to be read,
+see <<setting-variables>> to find out how to set the path the files.
 
 NOTE: Depending on how you've installed Filebeat, you might see errors
 related to file ownership or permissions when you try to run Filebeat modules.
 See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
 in the _Beats Platform Reference_ if you encounter errors related to file
 ownership or permissions.
 
-//include::system-module-note.asciidoc[]
-
-
-TIP: In a production environment, you'll probably want to use a configuration
-file, rather than command-line flags, to specify which modules to run. See the
-detailed documentation for more about configuring and running modules.
+This getting started guide uses the `-modules` flag to enable modules
+at the command line when you run Filebeat. In a production environment, you'll
+probably want to use the configs in the `modules.d` directory instead. See
+<<configuration-filebeat-modules>> for more information.
 
 [[setting-variables]]
 ==== Set the path variable
@@ -111,8 +122,8 @@ logs:
 ./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
 ----
 
-See the <<modules-tutorial>> for more information about setting variables and
-advanced options.
+See <<configuration-filebeat-modules>> for more information about setting
+variables and advanced options.
 
 [[passing-credentials-modules]]
 ==== Pass credentials

filebeat/docs/modules-overview.asciidoc

@@ -34,9 +34,20 @@ Node.
 
 Filebeat modules require Elasticsearch 5.2 or later.
 
+[float]
+=== Get started
+
+To learn how to configure and run Filebeat modules:
+
+* Get started by reading <<filebeat-modules-quickstart>>.
+* Try out the extended <<modules-tutorial>> for a more in-depth introduction.
+* Learn about the different ways to enable modules in <<configuration-filebeat-modules>>.
+
 [[modules-tutorial]]
 === Tutorial
 
+//REVIEWERS: Now that we have a section about specifying which modules to run (where I describe how to override settings) plus a Quick start for common log formats, the info in this tutorial is a bit redundant. I do think a more in-depth tutorial that shows a realistic use case would be pretty useful here. It could even show path overrides and prospector settings, but it wouldn't have to describe all the permutations because we do that in the reference section. IMO a streamlined version of this tutorial focused on specific user goals and built around a compelling story would be more useful. If someone can provide me with the "story" for this tutorial, I can write it up. Any volunteers?
+
 This tutorial assumes you have Elasticsearch and Kibana installed and
 accessible from Filebeat (see the <<filebeat-getting-started,getting started>>
 section). It also assumes that the Ingest Node GeoIP and User Agent plugins are
@@ -89,7 +100,8 @@ You can also start multiple modules at once:
 ./filebeat -e -modules=nginx,mysql,system
 ----------------------------------------------------------------------
 
-include::system-module-note.asciidoc[]
+
+//include::system-module-note.asciidoc[]
 
 While enabling the modules from the CLI file is handy for getting started and
 for testing, you will probably want to use the configuration file for the
@@ -148,6 +160,9 @@ cannot install the plugins, you can use the following:
 ./filebeat -e -modules=nginx -M "nginx.access.var.pipeline=no_plugins"
 ----------------------------------------------------------------------
 
+
+//REVIEWERS: Is no_plugins still a thing?
+
 ==== Advanced settings
 
 Behind the scenes, each module starts a Filebeat prospector. For advanced

libbeat/docs/command-reference.asciidoc

@@ -159,9 +159,11 @@ ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")]
 ==== `modules` command
 
 {modules-command-short-desc}. You can use this command to enable and disable
-specific modules. The changes you make with this command are persisted and
-used for subsequent runs of {beatname_uc}. To see which modules are enabled
-and disabled, run the `list` subcommand.
+specific module configurations defined in the `modules.d` directory. The
+changes you make with this command are persisted and used for subsequent
+runs of {beatname_uc}.
+
+To see which modules are enabled and disabled, run the `list` subcommand.
 
 *SYNOPSIS*