Add more context and examples to Filebeat module docs (#5341)

* Remove tutorial

* Add more context and examples to module docs

* Add fixes from review

filebeat/docs/include/config-option-intro.asciidoc

@@ -0,0 +1,16 @@
+[float]
+[id="{modulename}-settings"]
+==== Variable settings
+
+The +{modulename}+ module provides the following settings for configuring the
+behavior of the module. Each fileset has separate settings.
+
+If you don’t specify variable settings, the +{modulename}+ module uses the
+defaults.
+
+For more information, see <<specify-variable-settings>>. Also see
+<<advanced-settings>>.
+
+TIP: When you specify a setting at the command line, remember to prefix the
+setting with the module name, for example, +{modulename}.{fileset_ex}.var.paths+
+instead of +{fileset_ex}.var.paths+.

filebeat/docs/include/configuring-intro.asciidoc

@@ -0,0 +1,7 @@
+[float]
+[id="configuring-{modulename}-module"]
+=== Configure the module
+
+You can further refine the behavior of the +{modulename}+ module by specifying
+<<{modulename}-settings,variable settings>> in the
++modules.d/{modulename}.yml+ file, or overriding settings at the command line.

filebeat/docs/include/running-modules.asciidoc

@@ -0,0 +1,77 @@
+[float]
+[id="running-{modulename}-modules"]
+=== Set up and run the module
+
+IMPORTANT: If you’ve secured Elasticsearch and Kibana, you need to configure the
+`username` and `password` options in the Elasticsearch output before setting up
+and running the module. See
+<<elasticsearch-output,Configure the Elasticsearch output>>.
+
+Before doing these steps, verify that Elasticsearch and Kibana are running and
+that Elasticsearch is ready to receive data from {beatname_uc}.
+
+To set up and run the module:
+
+. Enable the module:
++
+["source","sh",subs="attributes"]
+----------------------------------------------------------------------
+./{beatname_lc} modules enable {modulename}
+----------------------------------------------------------------------
++
+The <<modules-command,`modules enable`>> command enables the +{modulename}+
+config defined in the `modules.d` directory. See
+<<configuration-{beatname_lc}-modules>> for other ways to enable modules.
++
+To see a list of enabled and disabled modules, run:
++
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} modules list
+-----
+
+
+. Set up the initial environment:
++
+["source","sh",subs="attributes"]
+----------------------------------------------------------------------
+./{beatname_lc} setup -e
+----------------------------------------------------------------------
++
+The <<setup-command,`setup`>> command loads the recommended index template for
+writing to Elasticsearch and deploys the sample dashboards for visualizing the
+data in Kibana. This is a one-time setup step. 
++
+The `-e` flag is optional and sends output to standard error instead of syslog.
+
+. Run {beatname_uc}:
++
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} -e
+-----
++
+If the module is configured correctly, you'll see
+`INFO Harvester started` messages for each file specified in the config.
++
+If your logs aren't in the default location, see
+<<configuring-{modulename}-module>>, then run {beatname_uc} after you've
+configured the module.
++
+NOTE: Depending on how you've installed Filebeat, you might see errors
+related to file ownership or permissions when you try to run Filebeat modules.
+See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
+in the _Beats Platform Reference_ if you encounter errors related to file
+ownership or permissions. 
+
+. Explore your data in Kibana:
+.. Open your browser and navigate to the *Dashboard* overview in Kibana:
+http://localhost:5601/app/kibana#/dashboards[http://localhost:5601/app/kibana#/dashboards].
+Replace `localhost` with the name of the Kibana host.
+.. If security is enabled, log in with the Kibana username and password that you
+used when you set up security.
+.. Enter *{modulename}* in the search box, then open a dashboard and explore
+the visualizations for your parsed logs.
++
+TIP: If you don’t see data in Kibana, try changing the date range to a larger
+range. By default, Kibana shows the last 15 minutes.

filebeat/docs/include/var-paths.asciidoc

@@ -0,0 +1,4 @@
+*`var.paths`*::
+
+An array of paths that specify where to look for the log files. If left empty,
+Filebeat will choose the paths depending on your operating systems.

filebeat/docs/include/what-happens.asciidoc

@@ -0,0 +1,12 @@
+When you run the module, it performs a few tasks under the hood:
+
+* Sets the default paths to the log files (but don't worry, you can override the
+defaults)
+
+* Makes sure each multiline log event gets sent as a single event
+
+* Uses ingest node to parse and process the log lines, shaping the data into a structure suitable
+for visualizing in Kibana
+
+* Deploys dashboards for visualizing the log data
+  

filebeat/docs/modules-overview.asciidoc

@@ -30,7 +30,10 @@ and loads them to the respective Elastic stack components.
 NOTE: At the moment, Filebeat modules require using the Elasticsearch
 {elasticsearch}/ingest.html[Ingest Node]. In the future, Filebeat Modules will
 be able to also configure Logstash as a more powerful alternative to Ingest
-Node.
+Node. For now, if you want to use Logstash, you can follow the steps described
+in the section called
+{logstashdoc}/filebeat-modules.html[Working with Filebeat Modules] in the
+Logstash Reference.
 
 Filebeat modules require Elasticsearch 5.2 or later.
 
@@ -40,150 +43,5 @@ Filebeat modules require Elasticsearch 5.2 or later.
 To learn how to configure and run Filebeat modules:
 
 * Get started by reading <<filebeat-modules-quickstart>>.
-* Try out the extended <<modules-tutorial>> for a more in-depth introduction.
 * Learn about the different ways to enable modules in <<configuration-filebeat-modules>>.
-
-[[modules-tutorial]]
-=== Tutorial
-
-//TODO: Replace this content with a more in-depth tutorial or remove it. 
-
-This tutorial assumes you have Elasticsearch and Kibana installed and
-accessible from Filebeat (see the <<filebeat-getting-started,getting started>>
-section). It also assumes that the Ingest Node GeoIP and User Agent plugins are
-installed. These plugins are required to capture the geographical location and
-browser information used by some of the visualizations available in the sample
-dashboards. You can install these plugins by running the following commands in the Elasticsearch home path:
-
-[source,shell]
-----------------------------------------------------------------------
-sudo bin/elasticsearch-plugin install ingest-geoip
-sudo bin/elasticsearch-plugin install ingest-user-agent
-----------------------------------------------------------------------
-
-You need to restart Elasticsearch after running these commands.
-
-If you are using an https://cloud.elastic.co/[Elastic Cloud] instance, you can
-enable the two plugins from the configuration page.
-
-This also assumes you have Nginx installed and writing logs in the default
-location and format. If you want to monitor another service for which a module
-exists, adjust the commands in the tutorial accordingly.
-
-You can start Filebeat with the following command:
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx -setup
-----------------------------------------------------------------------
-
-The `-e` flag tells Filebeat to output its logs to standard error, instead of
-syslog.
-
-The `--modules nginx` flag loads the Nginx module.
-
-The `-setup` flag tells Filebeat to load the associated sample Kibana
-dashboards. This setup phase, in which the dashboards are loaded, doesn't have
-to be executed each time, and because it's a relatively heavy operation, we
-recommend executing it only once after installing or upgrading Filebeat. That
-is why, the next commands from this tutorial are omitting the `-setup` flag.
-
-Visiting the Kibana web interface now, open the Nginx dashboard and you should
-already see your logs parsed and visualized in several widgets.
-
-image:./images/kibana-nginx.png[]
-
-You can also start multiple modules at once:
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx,mysql,system
-----------------------------------------------------------------------
-
-
-While enabling the modules from the CLI file is handy for getting started and
-for testing, you will probably want to use the configuration file for the
-production setup. The equivalent of the above in the configuration file is:
-
-
-[source,yaml]
-----------------------------------------------------------------------
-filebeat.modules:
-- module: nginx
-- module: mysql
-- module: system
-----------------------------------------------------------------------
-
-Then you can start Filebeat simply with: `./filebeat -e`.
-
-[[module-varialbe-overrides]]
-==== Variable overrides
-
-Each module and fileset has a set of "variables" which allow adjusting their
-behaviour. To see the available variables, you can consult the
-`filebeat.reference.yml` file. For example, all filesets allow setting a custom
-`paths` value, which is a list of Globs where the log files are searched.
-
-These variables have default values, sometimes depending on the operating
-system. You can override them either from the CLI via the `-M` flag, or from
-the configuration file.
-
-In the case of Nginx, for example, you can use the following if the access
-files are in a custom location:
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
-----------------------------------------------------------------------
-
-Or via the configuration file:
-
-[source,yaml]
-----------------------------------------------------------------------
-filebeat.modules:
-- module: nginx
-  access:
-    var.paths: ["/var/log/nginx/access.log*"]
-----------------------------------------------------------------------
-
-
-==== Advanced settings
-
-Behind the scenes, each module starts a Filebeat prospector. For advanced
-users, it's possible to add or overwrite any of the prospector settings. For
-example, enabling <<close-eof,close_eof>> can be done like this:
-
-
-[source,yaml]
-----------------------------------------------------------------------
-filebeat.modules:
-- module: nginx
-  access:
-    prospector:
-      close_eof: true
-----------------------------------------------------------------------
-
-Or like this:
-
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx -M "nginx.access.prospector.close_eof=true"
-----------------------------------------------------------------------
-
-From the CLI, it's possible to change variables or settings for multiple
-modules/fileset at once. For example, the following works and will enable
-`close_eof` for all the filesets in the nginx module:
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx -M "nginx.*.prospector.close_eof=true"
-----------------------------------------------------------------------
-
-The following also works and will enable `close_eof` for all prospectors
-created by any of the modules:
-
-[source,shell]
-----------------------------------------------------------------------
-./filebeat -e --modules nginx,mysql -M "*.*.prospector.close_eof=true"
-----------------------------------------------------------------------
+* Dive into the documentation for each module.

filebeat/docs/modules/apache2.asciidoc

@@ -3,11 +3,15 @@ This file is generated! See scripts/docs_collector.py
 ////
 
 [[filebeat-module-apache2]]
+:modulename: apache2
+
 == Apache2 module
 
-This module parses access and error logs created by the
+The +{modulename}+ module parses access and error logs created by the
 https://httpd.apache.org/[Apache HTTP] server.
 
+include::../include/what-happens.asciidoc[]
+
 [float]
 === Compatibility
 
@@ -16,40 +20,64 @@ This module requires the
 {elasticsearch-plugins}/ingest-geoip.html[ingest-geoip] Elasticsearch plugins.
 
 
-The Apache2 module was tested with logs from versions 2.2.22 and 2.4.23.
+The +{modulename}+ module was tested with logs from versions 2.2.22 and 2.4.23.
 
 On Windows, the module was tested with Apache2 installed from the Chocolatey
 repository.
 
-[float]
-=== Dashboard
+include::../include/running-modules.asciidoc[]
 
-This module comes with a sample dashboard.
+[float]
+=== Example dashboard
 
 image::./images/kibana-apache2.png[]
 
-[float]
-=== Access logs fileset settings
+include::../include/configuring-intro.asciidoc[]
 
-[float]
-==== var.paths
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for Apache2 access and error logs: 
+
+["source","yaml",subs="attributes"]
+-----
+- module: apache2
+  access:
+    enabled: true
+    var.paths: ["/path/to/log/apache2/access.log*"]
+  error:
+    enabled: true
+    var.paths: ["/path/to/log/apache2/error.log*"]
+-----
 
-An array of paths where to look for the log files. If left empty, Filebeat
-will choose the paths depending on your operating systems.
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} -M "apache2.access.var.paths=[/path/to/apache2/access.log*]" -M "apache2.error.var.paths=[/path/to/log/apache2/error.log*]"
+-----
+
+
+The command in the example assumes that you have already enabled the +{modulename}+ module. 
+
+//set the fileset name used in the included example
+:fileset_ex: access
+
+include::../include/config-option-intro.asciidoc[]
 
 [float]
-=== Error logs fileset settings
+==== `access` log fileset settings
+
+include::../include/var-paths.asciidoc[]
 
 [float]
-==== var.paths
+==== `error` log fileset settings
 
-An array of paths where to look for the log files. If left empty, Filebeat
-will choose the paths depending on your operating systems.
+include::../include/var-paths.asciidoc[]
 
 
 [float]
 === Fields
 
-For a description of each field in the metricset, see the
+For a description of each field in the module, see the
 <<exported-fields-apache2,exported fields>> section.