Improve user experience when using a custom ES index

[monicasarbu]

In case the user wants to use a custom index in Elasticsearch, we should provide a way for the user to configure the index name and have the Beat take care of adjusting the template, dashboards and the index where to insert the data.

What's available
Right now, if the user wants to use a custom index the following steps are done:

• edit the template to use the right ES index name
• during setup (-setup command line option), the user needs to specify the -E "dashboards.index=new_index" to adjust the Kibana index pattern and dashboards.

The issues

1. As of Dynamically generate template on startup #3681, the Beat will generate the template at startup, and there is not possible to edit the template anymore. So, we need to generate the template to include the right configured index pattern.

2. Where do we configure the ES index name? Right now, you can configure the custom index when loading the dashboards under dashboards.index, and it will change the index pattern and the dashboards. You can also configure the index where the data is stored in Elasticsearch:
   underoutput.elasticsearch.index in case ES output is enabled, under output.logstash.index in case of Logstash output.

In conclusion, there are different configuration options to configure the same index name. It would be nice to unify them into a single configuration option, maybe outside the output section.

cc-ed @ruflin @tsg

[ruflin]

I suggest we split this topic into two parts:

• Out of the box experience if elasticsearch is used as output
• Simplify experience when other outputs are used

For part 1, the dynamic loading should definitively take the index name into account and this should be done in #3681 or in a follow up PR.

For case 2, here some ideas:

Currently we have 3 things which for me fall under -setup:

• Loading index templates
• Loading Kibana dashboards and index patterns
• Loading ingest pipelines

All face the same issue in case elasticsearch is not defined as output. Our the monitoring we started to have a separate output section inside monitoring which takes the values from the output if not defined. We could the same for the above options. Something like:

setup:
  dashboard.*:
  template.*:
  ingest.*:
  elasticsearch.*:
  kibana.*:

For the setup parts, a separate output could be defined. This would also allow us for example to define a Kibana host to which data should be pushed. If a beat is run with -setup it uses the config options here. If no other output or config options are set -setup could also be used with only the setup part in the config and would stop afterwards.

We should build an extensible solution here as potentially more setup parts will be added in the future. The above also does not require us to be able to send all this information through Logstash, but one beat could be configured locally that has direct access to elasticserach.

[monicasarbu]

This meta ticket is for tracking all the changes that we need to do in order to improve the user experience during the setup phase when he/she wants to customize the index name, no matter what output he/she is configuring. I didn't intend to include how we can improve loading templates, dashboards, etc when another output is configured, different than ES. I agree that this should be a different meta-ticket. For now, the user needs to call the Beat with the -setup option on a different host where he has direct access to the ES cluster.

[Porubay]

I don`t have much to say about it, but I know where to read about a little part of it.

[adoerrES]

I linked a new entry to improve the documentation around required steps for using custom index patterns with the example Dashboards.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.