Simplify Kibana Files (Dashboard) Handling

[ruflin]

Dashboards, visualisations and search are a central part of each beat as it is critical to not only collect and store the data, but also visualise it. Currently the beats-dashboards with the Kibana files life in their own repository: https://github.com/elastic/beats-dashboards This makes it kind of hard to maintain and install for new users, as they have to download additional files.

The following steps should be taken to simplify the handling of dashboards:

• Move Kibana files to their specific beat Add dashboards to winlogbeat repo #1103 Simplify dashboard handling for developers #1109
• Implement Kibana file loading in libbeat (similar to template loading)
• Move export / loading of Kibana files to libbeat so also community beats can use it
• Update deployment scripts to include kibana files.
• Move docs to beats docs including screenshots

[monicasarbu]

@ruflin I agree that it would be better to have the Kibana dashboard together with the visualizations, searches and index patterns in the Beat itself and not in a different repository. I do think we should find a way to load and export the files using Kibana and not a script. In my opinion the solution that we have with the bash/python scripts is temporary until we have a better solution in Kibana (that doesn't depend on the OS).

cc-ed @tsg

[ruflin]

@monicasarbu Fully agree. It would be best if in the long term Kibana would even ship already with these templates (same as LS does with the beats plugin). Until we are there, my goal is that we automate it further by implementing this capability into libbeat, so no shell scripts are needed.

[monicasarbu]

@ruflin After playing a bit more with the export/import functionality in Kibana, I realized that there are a few changes that are particular useful only for Beats like exporting/importing the index patterns, dealing with the default search (to a logstash-* index that doesn't exist). I agree with the idea to have something automated in the Beat itself that does everything for the user.

[ruflin]

@monicasarbu Can you share some more insigths into your findings with the importing / exporting?

[monicasarbu]

@ruflin If you "Export Everything" in Kibana, you get a single json file that contains the dashboards, visualizations and searches, but not the index patterns. If you are using our python script then the index patterns are also downloaded.

I noticed couple of problems with the current dashboards that we have, while importing/exporting our dashboards:

• we need to offer for download only our dashboards/visualizations/searches and remove Default Search for example (I know there were a few discussions about it)
• there are few searches that are missing like DB-transactions, Cache-transactions, so we need to make sure the Packetbeat visualizations are using existing searches.

Kibana is a great way of exporting/importing as they have all these checks and the users make sure the dashboards are completely defined (including the visualizations and searches), but it's missing a way to export/import the index-patterns. We should add a similar check when the Beat is loading the dashboards (in #1103).

[ruflin]

@monicasarbu Thanks for the details. I had a look at the json file exported by Kibana and it "only" nests all document inside a json array. It would be awesome if Kibana could expose the import feature including checks through an API. Kibana already has an issue for that: https://github.com/elastic/kibana/issues/5431It is about exporting/importing the complete index. I would hope that this also includes checks.

[monicasarbu]

Closing this as some of the points were already solved in other PRs.