Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import "beats" dashboards #247

Merged
merged 7 commits into from
Mar 11, 2020
Merged

Import "beats" dashboards #247

merged 7 commits into from
Mar 11, 2020

Conversation

mtojek
Copy link
Contributor

@mtojek mtojek commented Mar 11, 2020

This PR extends "import-beats" script to pull dashboards and visualizations.

I think we can iterate on top (index names, etc.) of this to come a final version of Kibana artifacts.

Issue: #221

@mtojek mtojek requested a review from ruflin March 11, 2020 11:33
@mtojek mtojek self-assigned this Mar 11, 2020
//
// If you need insert keys containing dots then you must use bracket notation
// to insert values (e.g. m[key] = value).
func (m mapStr) put(key string, value interface{}) (interface{}, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso We should really put mapStr in its own packages. I also had other uses in the past for it.

Copy link
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Comments can be in follow up PRs.

If I understand it correctly, you load the dashboards into KB and then pull them out again?

}

request, err := http.NewRequest("POST",
fmt.Sprintf("http://%s/api/kibana/dashboards/import?force=true", km.hostPort),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is force doing here? Just curious

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query parameters

force
(Optional, boolean) Overwrite any existing objects on ID conflict.

if os.IsNotExist(err) {
log.Printf("\tno dashboards present, skipped (modulePath: %s)", modulePath)
return kibanaContent{}, nil
} else if err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Problably no else needed as you return in the if clause above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

return nil, errors.Wrapf(err, "retrieving id failed")
}

extracted[id.(string)+".json"] = data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The id of the saved object becomes the name? I like that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true :)

"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[],\"query\":{\"language\":\"kuery\",\"query\":\"\"}}"
},
"optionsJSON": "{\"hidePanelTitles\":false,\"useMargins\":true}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JSON strings should be decoded inside the package. I think you have all the logic for it already in place for the encoding, otherwise see #214

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... if you want to upload them to Kibana, you need to encode them again. I was wondering if it's not better to keep them encoded in packages as well. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we decode them is for versioning purpose. If you change anything small int he visualisation and then get a PR with the change, it is almost impossible to figure out what change. That is the reason we switched to the decoded format in Beats some while ago and I would like also to use it here. My hope is that in the future Kibana directly supports the decoded option.

The encoding currently happens in the build step.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you're right! Versioning is a killer argument in this thread.

I've just pushed a change that decodes fields.

@mtojek
Copy link
Contributor Author

mtojek commented Mar 11, 2020

LGTM. Comments can be in follow up PRs.

If I understand it correctly, you load the dashboards into KB and then pull them out again?

I think there is no need to pull them out. Once you POST them to Kibana, in the response you receive migrated/converted objects.

@mtojek mtojek mentioned this pull request Mar 11, 2020
@ruflin
Copy link
Contributor

ruflin commented Mar 11, 2020

That you get it in the response was the piece I was missing. This simplifies things!

@mtojek mtojek merged commit af697bf into elastic:master Mar 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants