Use Saved Objects API for exporting and importing dashboards #27220

kvch · 2021-08-03T16:19:04Z

What does this PR do?

This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations:

You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats.
You can only export new assets, not legacy format.

TODO

migrate dashboards of Winlogbeat
add a few sections to Beats developer guide to explain how to migrate existing assets
revisit modify JSON functions
follow up in dashboards end to end test

Why is it important?

We have to move forward to let Kibana remove the old API.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Try to setup the dashboards of a Beat:

./filebeat setup --dashboards

Export dashboards:

./filebeat export dashboard --id {my-id}

and:

go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id}

Notes to the reviewer

The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson.

Please also test it manually with both the setup command and export_dashboards.go.

Related issues

Closes #20672

elasticmachine · 2021-08-03T16:44:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-13T15:52:41.507+0000
Duration: 160 min 59 sec
Commit: 5541d2d

Test stats 🧪

Test	Results
Failed	0
Passed	53106
Skipped	5320
Total	58426

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53106
Skipped	5320
Total	58426

elasticmachine · 2021-08-05T16:37:03Z

Pinging @elastic/agent (Team:Agent)

exekias · 2021-08-09T07:18:56Z

@mtojek once this goes in it will affect the beats import script, as this is already converting dashboards. On the other side, I don't think we are using the same format, thoughts?

kvch · 2021-08-09T11:32:13Z

IIUC this means that Beats 7.15 will no longer import dashboards to 7.14. Did I get that right? If so does that match with what's in the support matrix? I think this it would be helpful to call that out in the changelog/release notes.

I gave the Go code a quick pass. I didn't try running it.

I have added it to our development documentation. Let me know if that is not clear. Also, I am adding it to the changelog soon.

andrewkroh · 2021-08-09T11:37:22Z

I have added it to our development documentation. Let me know if that is not clear. Also, I am adding it to the changelog soon.

That sounds good. Any idea about the support matrix? I'm having a hard time interpreting what compatibility guarantees are made there.

kvch · 2021-08-09T12:59:21Z

I have added it to our development documentation. Let me know if that is not clear. Also, I am adding it to the changelog soon.

That sounds good. Any idea about the support matrix? I'm having a hard time interpreting what compatibility guarantees are made there.

Well, what I inferred from the support matrix is that version(Beats) <= version(Elasticsearch): https://www.elastic.co/support/matrix#matrix_compatibility

This is also what we agreed on in: #20672 (comment)

kvch · 2021-08-11T15:47:19Z

jenkins run tests

kvch · 2021-08-12T08:04:19Z

jenkins run tests

andrewkroh · 2021-08-13T16:06:38Z

libbeat/dashboards/decode.go

+	}
+}
+
+func decodeLine(line []byte) []byte {


I think json.Unmarshal will return an error if you try to decode an empty slice.

You might want to do a if len(bytes.TrimSpace(line)) == 0 { return }. This handles both empty slices and blank lines (which are allowable per the NDJSON spec you linked to.)

👍 I have added this check to more functions that needed it.

kvch · 2021-08-16T14:19:14Z

Failing test is unrelated.

This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. We have to move forward to let Kibana remove the old API. Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. Closes #20672

…#27398) This PR adopts the Saved Objects API for exporting and importing dashboards and index templates. We have agreed to provide essential support, so there are a few limitations: * You can only import assets they have exported using Beats 7.15 and the assets we ship with Beats. * You can only export new assets, not legacy format. We have to move forward to let Kibana remove the old API. Try to setup the dashboards of a Beat: ```sh ./filebeat setup --dashboards ``` Export dashboards: ```sh ./filebeat export dashboard --id {my-id} ``` and: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {my-id} ``` The changeset is huge, but most of it comes from converting the dashboards. I suggest you use the file filter and disable json, ndjson and yml files. The changes are moving json to ndjson format and changing the extensions in the module.yml files from json to ndjson. Please also test it manually with both the setup command and `export_dashboards.go`. Closes #20672 Co-authored-by: Noémi Ványi <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 3, 2021

kvch added 5 commits August 4, 2021 19:58

Initialize exporting dashboards

d584667

tmp

b5d8612

tmp

e9c0418

cleanup

172ca9d

follow-up tests

ef2cc98

kvch force-pushed the feature-libbeat-import-objects branch from e8e9a47 to 1d96f7f Compare August 4, 2021 17:59

kvch added 4 commits August 4, 2021 20:06

rm old dashboards

459e3de

more fixes

23c97d7

more follow-up

fc404c0

tmp

2d44a6c

kvch force-pushed the feature-libbeat-import-objects branch from 1d96f7f to 2d44a6c Compare August 4, 2021 18:07

kvch requested a review from urso August 4, 2021 18:14

kvch changed the title ~~Feature libbeat import objects~~ Use Saved Objects API for exporting and importing dashboards Aug 4, 2021

kvch added the Team:Elastic-Agent Label for the Agent team label Aug 4, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 4, 2021

kvch added 8 commits August 5, 2021 09:25

fix go vet

3f9a156

follow up in test

e93c8d3

add docs

773ba59

update reference in module files

08b925c

more more

d1c4067

fix e2e tests

88c52b3

follo up more

8ec3b6e

one more

def1720

kvch marked this pull request as ready for review August 5, 2021 16:37

kvch requested a review from a team as a code owner August 5, 2021 16:37

kvch added the backport-v7.15.0 Automated backport with mergify label Aug 5, 2021

go vet

23f47fe

address review notes

60642e8

kvch added 6 commits August 9, 2021 15:16

follow up tests

ffcc7a3

follow up tests

53072cc

Skip test if Kibana is not required version

978336e

create separate value for minimum required version

ff1ca85

add missing winlogbeat dashboards

eec349e

add changelog entry

85cd464

lukeelmers mentioned this pull request Aug 11, 2021

Remove support for legacy imports from the UI elastic/kibana#103921

Closed

3 tasks

parse last line

5541d2d

andrewkroh reviewed Aug 13, 2021

View reviewed changes

add check for empty lines

513eedf

andrewkroh approved these changes Aug 16, 2021

View reviewed changes

kvch merged commit 099dfa3 into elastic:master Aug 16, 2021

mergify bot mentioned this pull request Aug 16, 2021

[7.x](backport #27220) Use Saved Objects API for exporting and importing dashboards #27398

Merged

lukeelmers mentioned this pull request Aug 16, 2021

Remove Dashboard Import/Export API elastic/kibana#41439

Open

6 tasks

jsoriano mentioned this pull request Aug 17, 2021

mage importBeats fails to import dashboards in ndjson format elastic/integrations#1536

Closed

endorama mentioned this pull request Aug 17, 2021

[Metricbeat] CI: update stack to 7.14 #27416

Merged

6 tasks

liza-mae mentioned this pull request Aug 17, 2021

Import beats dashboard fails with unsupported media type #27418

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Saved Objects API for exporting and importing dashboards #27220

Use Saved Objects API for exporting and importing dashboards #27220

kvch commented Aug 3, 2021 •

edited

Loading

elasticmachine commented Aug 3, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Aug 5, 2021

exekias commented Aug 9, 2021

kvch commented Aug 9, 2021

andrewkroh commented Aug 9, 2021

kvch commented Aug 9, 2021

kvch commented Aug 11, 2021

kvch commented Aug 12, 2021

andrewkroh Aug 13, 2021

kvch Aug 16, 2021

kvch commented Aug 16, 2021

Use Saved Objects API for exporting and importing dashboards #27220

Use Saved Objects API for exporting and importing dashboards #27220

Conversation

kvch commented Aug 3, 2021 • edited Loading

What does this PR do?

TODO

Why is it important?

Checklist

How to test this PR locally

Notes to the reviewer

Related issues

elasticmachine commented Aug 3, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Aug 5, 2021

exekias commented Aug 9, 2021

kvch commented Aug 9, 2021

andrewkroh commented Aug 9, 2021

kvch commented Aug 9, 2021

kvch commented Aug 11, 2021

kvch commented Aug 12, 2021

andrewkroh Aug 13, 2021

Choose a reason for hiding this comment

kvch Aug 16, 2021

Choose a reason for hiding this comment

kvch commented Aug 16, 2021

kvch commented Aug 3, 2021 •

edited

Loading

elasticmachine commented Aug 3, 2021 •

edited by jenkins-beats-ci bot

Loading