Skip to content

Commit

Permalink
Add docs for ILM feature (#9263)
Browse files Browse the repository at this point in the history
* Add docs for ILM feature

This is a follow up for #7963

* Add edits

* Minor fixes

* Add changes from the review

* Make cross references consistent

* Fix rebase error
  • Loading branch information
ruflin authored and dedemorton committed Jan 22, 2019
1 parent 84844b3 commit 2bd4f50
Show file tree
Hide file tree
Showing 10 changed files with 182 additions and 1 deletion.
3 changes: 3 additions & 0 deletions auditbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ The following topics describe how to configure {beatname_uc}:
* <<{beatname_lc}-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand Down Expand Up @@ -53,6 +54,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./auditbeat-filtering.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions filebeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The following topics describe how to configure Filebeat:
* <<filebeat-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<load-balancing>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand Down Expand Up @@ -61,6 +62,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ilm.asciidoc[]

include::./load-balancing.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions heartbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ The following topics describe how to configure Heartbeat:
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand All @@ -50,6 +51,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./heartbeat-filtering.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions journalbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The following topics describe how to configure {beatname_uc}:
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand All @@ -39,6 +40,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./filtering.asciidoc[]
Expand Down
15 changes: 14 additions & 1 deletion libbeat/docs/outputconfig.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,17 @@ output.elasticsearch:
protocol: "https"
username: "{beatname_lc}_internal"
password: "{pwd}"
------------------------------------------------------------------------------


For more information about securing {beatname_uc}, see
<<securing-{beatname_lc}>>.

If you are indexing large amounts of time-series data, you might also want to
configure {beatname_uc} to use index lifecycle management. For more information
about configuring and using index lifecycle management with {beatname_uc}, see
<<ilm>>.

==== Compatibility

This output works with all compatible versions of Elasticsearch. See the
Expand Down Expand Up @@ -338,6 +342,15 @@ This configuration results in indices named `sev1`, `sev2`, and `sev3`.
The `mappings` setting simplifies the configuration, but is limited to string
values. You cannot specify format strings within the mapping pairs.

//TODO: MOVE ILM OPTIONS TO APPEAR LOGICALLY BASED ON LOCATION IN THE YAML FILE.

[[ilm-es]]
===== `ilm`

Configuration options for index lifecycle management.

See <<ilm>> for more information.

ifndef::no-pipeline[]
[[pipeline-option-es]]
===== `pipeline`
Expand Down
143 changes: 143 additions & 0 deletions libbeat/docs/shared-ilm.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
[[ilm]]
[role="xpack"]
== Set up index lifecycle management

beta[]

You can use the {ref}/getting-started-index-lifecycle-management.html[index
lifecycle management] feature in {es} to manage your {beatname_uc} indices as
they age. For example, instead of having {beatname_uc} create daily indices
where index size can vary based on the number of Beats and number of events
sent, you can use an index lifecycle policy that automates a rollover to a new
index when the existing index reaches a specified size or age.

{beatname_uc} provides a default policy that you can load when you set up
{beatname_uc}. The default policy is applied to any new indices created by
{beatname_uc}. You can edit the policy to modify the lifecycle of both new and
existing indices.

To use index lifecycle management on {beatname_uc} indices:

. Enable index lifecycle management by setting `ilm.enabled: true` in the {es}
output configuration. For example:
+
--
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
ilm.enabled: true
------------------------------------------------------------------------------

This configuration overwrites your index settings and adjusts the {beatname_uc}
template to use the correct settings for index lifecycle management.

NOTE: If you've previously loaded the index template for this version into {es},
you must overwrite the template by setting `setup.template.overwrite: true`.

The rollover alias is set to +{beatname_lc}-\{beat.version\}+ by default. You
can change the prefix used in the alias by setting `ilm.rollover_alias`, but you
can't remove `{beat.version}` from the rollover alias name. The default pattern
used for the rollover index is `%{now/d}-000001`. You can change the
pattern by setting `ilm.pattern`. For example:

["source","yaml",subs="attributes"]
----
output.elasticsearch:
hosts: ["localhost"]
ilm.enabled: true
ilm.rollover_alias: "{beatname_lc}"
ilm.pattern: "{now/d}-000001" <1>
----
<1> Date math is supported here. For more information, see
{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Using
date math with the rollover API].

NOTE: If you modify the `rollover_alias` or `pattern` settings after loading the
index template, you must overwrite the template to apply the changes.
--

. Load the default policy into {es}. You can either use the `setup` command to
load the policy without modifying it, or modify the policy and load it manually.
+
--
To use the setup command, run:

["source","shell",subs="attributes"]
----
{beatname_lc} setup --ilm-policy
----

After loading the default policy, you can edit it in the *Index lifecycle policies*
UI in {kib}. For more information about working with the UI, see
{kibana-ref}/index-lifecycle-policies.html[Index lifecyle policies].

To modify the default policy before loading it, run +{beatname_lc} export
ilm-policy+ to print the policy to stdout. Modify the policy then use the
{ref}/ilm-put-lifecycle.html[Create lifecycle policy API] to load it into {es}.

--

[float]
=== Advanced ILM settings

WARNING: We recommend that you avoid modifying these settings unless you know
what you're doing.

The default index lifecycle management settings work best for common use cases
that work with the automated alias setup described earlier. It is possible to
use a multiple write alias with dynamic index patterns, but this requires manual
set up. This section describes the configuration options you need to change.

Let's assume you have the index pattern `customname-%{event.module}` where
`event.module` can have the values `system` and `apache`. First you must set up
a rollover index for `customname-system` and `customname-apache`. For details on
how to do this, see
{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
Index].

Next, set the index pattern in the {es} output. For example:

["source","yaml",subs="attributes"]
----
output.elasticesarch.index: customname-%{event.module} <1>
----
<1> For this example to work, every event must contain `event.module`.

If you change the index name, you must also set the template name, template
pattern, rollover alias, and lifecycle name. The best way to set these is
through an {es} template. It's possible to disable the template loading in
{beatname_uc} and specify these settings in your own template. Or you can use
the following config options in {beatname_uc}:

[source,yaml]
----
setup.template.name: "customname"
setup.template.pattern: "customname-*"
setup.template.settings.index.lifecycle.rollover_alias: "customname"
setup.template.settings.index.lifecycle.name: "beats-default-policy"
----

IMPORTANT: If you set the options manually as shown in this example, do *not*
set `ilm.enabled`, or the settings specified in the configuration file will be
overwritten.

This configuration results in a managed index named something like
+customname-{localdate}-000001+ and the following index settings:

["source","shell"]
----
"aliases" : {
"customname" : {
"is_write_index" : true
}
},
...
"index" : {
"lifecycle" : {
"name" : "beats-default-policy",
"rollover_alias" : "customname"
},
----


3 changes: 3 additions & 0 deletions metricbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The following topics describe how to configure {beatname_uc}:
* <<metricbeat-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand Down Expand Up @@ -53,6 +54,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./metricbeat-filtering.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions packetbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The following topics describe how to configure Packetbeat:
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand All @@ -50,6 +51,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./packetbeat-filtering.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions winlogbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The following topics describe how to configure Winlogbeat:
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand All @@ -44,6 +45,8 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]

include::{libbeat-dir}/docs/outputconfig.asciidoc[]

include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

include::./winlogbeat-filtering.asciidoc[]
Expand Down
4 changes: 4 additions & 0 deletions x-pack/functionbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The following topics describe how to configure {beatname_uc}:
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<ilm>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
* <<configuring-ingest-node>>
Expand All @@ -41,6 +42,9 @@ include::{libbeat-dir}/docs/queueconfig.asciidoc[]
[role="xpack"]
include::{libbeat-dir}/docs/outputconfig.asciidoc[]

[role="xpack"]
include::{libbeat-dir}/docs/shared-ilm.asciidoc[]

[role="xpack"]
include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]

Expand Down

0 comments on commit 2bd4f50

Please sign in to comment.