Use import/export API for loading dashboards and index pattern #4413

monicasarbu · 2017-05-29T14:05:24Z

This is trying to implement #4409.

The Beats 6.x dashboards are different than the Beats 5.x dashboards and are not compatible. Thanks to the Kibana API, we can have a json file for each module instead of multiple json files (for dashboards, visualizations, searches, etc). Thus, the way to import/export dashboards differs from the Kibana version that should be equal to the Elasticsearch version.

The json file containing the index-pattern contain as version the Beat version, and the dashboards contain as version the Kibana version that exported the dashboards.

If Kibana version is 5.x and older:

The dashboards should be imported/exported directly from Elasticsearch, and it works only if Elasticsearch is set as output
All the Beats dashboards are moved to the _meta/kibana/5.x directory

If Kibana version is 6.x and greater:

The dashboards should be imported/exported via the Kibana API, so the user needs to configure the kibana url under the setup.kibana* settings.
All the Beats 6.x dashboards should be added to the _meta/kibana/default directory for each Beat module.

In order to make use of the import/export API the following changes are needed:

move all the old dashboards, search, visualizations, index-pattern to _meta/kibana/5.x directory
and create a new directory under _meta/kibana/default for the new dashboards and index-pattern that are importable via the Kibana API
update generate_index_pattern to generate the index-pattern for 5.x and >= 6.x versions
use the Kibana API to load the dashboards and the index pattern for Kibana >= 6
update import_dashboards script
update make update
update integration tests

Note: The older dashboards are moved to the _meta/kibana/5.x directory only for Metricbeat for now.

ruflin · 2017-05-30T03:57:18Z

@monicasarbu Should this also go under dev-tools/cmd/* like all the other commands?

monicasarbu · 2017-06-01T11:47:11Z

@ruflin This PR is at a very early stage. My plan is to move the import and export functionality to libbeat and then create a script to use it. For now it's just a script (that I will remove) only for testing the import/export API.

tsg · 2017-06-28T12:25:58Z

I think we need to add setup.kibana to the short config as well, using localhost by default.

monicasarbu · 2017-06-28T12:58:31Z

@tsg Fixed, thanks!

tsg · 2017-06-28T13:47:33Z

libbeat/dashboards/dashboards.go

+		return fmt.Errorf("fail to create the Kibana loader: %v", err)
+	}
+
+	if kibanaLoader == nil {


Seems that etiher err == nil or kibanaLoader != nil, so I think the second check here is not needed and it doesn't follow our usual error checking pattern.

tsg · 2017-06-28T13:52:36Z

libbeat/dashboards/kibana_loader.go

+}
+
+func (loader KibanaLoader) Close() error {
+	//return loader.client.Close()


Seems that there's nothing to do on Close. Then you can just delete the comment, not to confuse future readers.

tsg · 2017-06-28T13:54:16Z

libbeat/setup/kibana/client.go

+	logp.Debug("kibana", "Kibana url: %s", kibanaURL)
+
+	client := &Client{
+


Minor issue, but this extra empty line could be deleted.

tsg · 2017-06-28T13:57:00Z

libbeat/setup/kibana/client.go

+			URL: kibanaURL,
+			http: &http.Client{
+				Transport: &http.Transport{
+					TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // ignore expired SSL certificates


Hmm, does this mean that HTTPS is always enabled? The comment looks like "insecure by default", we have to be careful with this.

This is something that we need to address in the next PR.

Added to the TODO list: #4409

tsg · 2017-06-28T20:23:23Z

libbeat/dashboards/kibana_loader.go

+
+	if cfg == nil {
+		cfg = common.NewConfig()
+		err := cfg.SetBool("enabled", -1, true)


"enabled" is forced to true here, but I didn't see where it's actually checked. We should make sure that disabling with enabled: false works.

tsg · 2017-06-28T20:24:31Z

libbeat/setup/kibana/client.go

+
+	kibanaURL, err := common.MakeURL(config.Protocol, config.Path, config.Host, 5601)
+	if err != nil {
+		logp.Err("Invalid host param set: %s, Error: %v", config.Host, err)


We generally avoid logp.Err immediately followed by returning an error.

tsg · 2017-06-28T20:37:03Z

libbeat/setup/kibana/client.go

+
+	reqURL, err := addPath(conn.URL, extraPath)
+	if err != nil {
+		return 0, nil, fmt.Errorf("fail to parse URL %s: %v", conn.URL, err)


Would be perhaps better to write the "fail to parse URL" error in the addPath method?

tsg · 2017-06-28T20:37:30Z

libbeat/setup/kibana/client.go

+
+	req, err := http.NewRequest(method, reqURL, body)
+	if err != nil {
+		logp.Err("Fail to create a HTTP %s request: %v", method, err)


Return error instead of logging.

tsg · 2017-06-28T20:38:07Z

libbeat/setup/kibana/client.go

+
+	resp, err := conn.http.Do(req)
+	if err != nil {
+		logp.Err("Fail to execute a HTTP %s request: %v", method, err)


tsg · 2017-06-28T20:39:55Z

libbeat/setup/kibana/client.go

+
+	_, result, err := client.Connection.Request("GET", "/api/status", nil, nil)
+	if err != nil {
+		return "", fmt.Errorf("HTTP GET request to /api/status fails: %v", err)


Please add the result to the error message.

tsg · 2017-06-28T20:40:32Z

libbeat/setup/kibana/client.go

+	var kibanaVersion kibanaVersionResponse
+	err = json.Unmarshal(result, &kibanaVersion)
+	if err != nil {
+		return "", fmt.Errorf("fail to unmarshall the HTTP response from Kibana %s: %v", client.Connection.URL, err)


unmarshal with one l.

tsg · 2017-06-28T20:41:28Z

libbeat/setup/kibana/client.go

+	if err != nil {
+		return fmt.Errorf("%v. Response: %s", err, response)
+	}
+	if statusCode != 200 {


200 might be too strict here. Maybe > 300?

true, fixed

tsg · 2017-06-28T20:46:41Z

libbeat/dashboards/kibana_loader.go

+)
+
+var importAPI = "/api/kibana/dashboards/import"
+var exportAPI = "/api/kibana/dashboards/export"


exportAPI doesn't seem to be used.

tsg · 2017-06-28T21:35:31Z

libbeat/scripts/Makefile

@@ -305,16 +306,8 @@ index-template: ## @build Generate index templates for the given $VERSION. This
 ### KIBANA FILES HANDLING ###
 ES_URL?=http://localhost:9200

-.PHONY: export-dashboards
-export-dashboards: python-env update
-	. ${PYTHON_ENV}/bin/activate && python ${ES_BEATS}/dev-tools/export_dashboards.py --url ${ES_URL} --dir $(shell pwd)/_meta/kibana --regex ${BEAT_NAME}-*


Could we get export-dashboards and import-dashboards targets based on the go code? I'm good with follow up PR for them.

It's a bit tricky for export-dashboards as we need to pass the list of ids that we want to export in one json file. Usually you would run it to export the dashboards for a module, not for a Beat. Added to the TODO list in the meta issue: #4409

exekias · 2017-06-29T14:53:53Z

I think it would make sense to add dashboards to the existing export command, I think users could use that to share easily their own dashboards

ruflin

If we update / improve dashboards for 6.x in the future, will we also update 5.x dashboards? Probably best would be that we "freeze" them as they are and improvements / new features only are available for 6.x?

ruflin · 2017-07-02T08:56:17Z

.gitignore

@@ -22,6 +22,9 @@ coverage.out
 *.swo
 *.swn

+# Executable


I would move this to a local ignore file in dashboards

Good idea, done.

ruflin · 2017-07-02T08:56:45Z

auditbeat/auditbeat.reference.yml

+  # Scheme and port can be left out and will be set to the default (http and 5601)
+  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
+  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
+  # host: "localhost:5601"


Remove spaces in front of config options for easier uncommenting. Also below.

ruflin · 2017-07-02T08:59:33Z

libbeat/dashboards/dashboards.go

+}
+func getMajorVersion(version string) (int, error) {
+
+	majorVersion := string(version[0])


This will break with release 10.* :-)

monicasarbu · 2017-07-03T08:24:09Z

@ruflin yes, that's the plan to freeze the 5.x dashboards and improve only the 6.x dashboards.

If the Elasticsearch version is older than 6.x, then the dashboards are loaded directly into Elasticsearch. The old dashboards are moved to the 5.x directory.

tsg · 2017-07-03T13:23:55Z

jenkins, test it

monicasarbu added the in progress Pull request is currently in progress. label May 29, 2017

monicasarbu changed the title ~~Use import/export API for loading dashboards and index pattern~~ [WIP] Use import/export API for loading dashboards and index pattern Jun 1, 2017

monicasarbu force-pushed the use_import_export_api branch from e85f3e2 to 8e271db Compare June 12, 2017 11:20

monicasarbu added the :Dashboards label Jun 15, 2017

monicasarbu force-pushed the use_import_export_api branch 2 times, most recently from 9160e6f to a294e57 Compare June 24, 2017 07:07

monicasarbu added the review label Jun 24, 2017

monicasarbu changed the title ~~[WIP] Use import/export API for loading dashboards and index pattern~~ Use import/export API for loading dashboards and index pattern Jun 24, 2017

monicasarbu force-pushed the use_import_export_api branch 5 times, most recently from 0efa466 to 9446eff Compare June 28, 2017 11:48

monicasarbu removed the in progress Pull request is currently in progress. label Jun 28, 2017

tsg reviewed Jun 28, 2017

View reviewed changes

monicasarbu mentioned this pull request Jun 29, 2017

Import/export dashboards via the Kibana API #4409

Closed

12 tasks

monicasarbu force-pushed the use_import_export_api branch 2 times, most recently from 00523db to 6139a64 Compare June 29, 2017 13:04

monicasarbu force-pushed the use_import_export_api branch from e60dafc to afaaeeb Compare June 29, 2017 16:53

tsg mentioned this pull request Jun 30, 2017

Remove import_dashboards from packaging #4586

Merged

monicasarbu mentioned this pull request Jun 30, 2017

Update screenshot for the system module #4589

Merged

ruflin reviewed Jul 2, 2017

View reviewed changes

monicasarbu force-pushed the use_import_export_api branch from a5b8147 to 1496049 Compare July 3, 2017 08:19

monicasarbu force-pushed the use_import_export_api branch from 218c11b to 57d4a31 Compare July 3, 2017 09:46

monicasarbu added 2 commits July 3, 2017 12:53

Load dashboards using the Kibana API

e5624b5

If the Elasticsearch version is older than 6.x, then the dashboards are loaded directly into Elasticsearch. The old dashboards are moved to the 5.x directory.

Fix the CPU usage visualization

25f5c73

monicasarbu force-pushed the use_import_export_api branch from c0f1f8b to 25f5c73 Compare July 3, 2017 10:53

monicasarbu mentioned this pull request Jul 3, 2017

Update dashboards for Packetbeat, Heartbeat and Winlogbeat #4594

Merged

monicasarbu mentioned this pull request Jul 3, 2017

Add auth support in Kibana #4597

Merged

tsg merged commit 4c9267e into elastic:master Jul 3, 2017

tsg mentioned this pull request Jul 24, 2017

Document breaking changes in 6.0 #4737

Closed

28 tasks

tylersmalley mentioned this pull request Dec 1, 2017

Importing dashboards from icingabeat renders kibana 6.0 unusable elastic/kibana#15339

Closed

		logp.Debug("kibana", "Kibana url: %s", kibanaURL)

		client := &Client{

Use import/export API for loading dashboards and index pattern #4413

Use import/export API for loading dashboards and index pattern #4413

Conversation

monicasarbu commented May 29, 2017 • edited Loading

ruflin commented May 30, 2017

monicasarbu commented Jun 1, 2017

tsg commented Jun 28, 2017

monicasarbu commented Jun 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exekias commented Jun 29, 2017

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

monicasarbu commented Jul 3, 2017

tsg commented Jul 3, 2017

monicasarbu commented May 29, 2017 •

edited

Loading