Basic server side export API for saved objects #30326

mikecote · 2019-02-06T22:07:59Z

PR implements the first steps of the saved object export API for #4759.

An endpoint id added under POST /api/saved_objects/_export that throws an error after 10,000 objects. The removal of the limit will come in a future PR when using streams to optimize scalability and performance. This basic PR will allow focus on the import aspect sooner as that will be more complex to solve.

Also a future PR will change the management UI to use the new API once import and export APIs are complete on the server side.

Dev-Docs

Added server side API for exporting saved objects

A new endpoint /api/saved_objects/_export now exists to export saved objects.

elasticmachine · 2019-02-06T22:08:01Z

Pinging @elastic/kibana-platform

…basic-export

src/server/saved_objects/routes/export.js

…basic-export

src/server/saved_objects/lib/export.ts

…basic-export

Co-Authored-By: mikecote <[email protected]>

spalger

LGTM!

epixa · 2019-03-04T01:59:03Z

Why have two separate endpoints rather than a single endpoint with an optional types argument? The rationale might exist somewhere buried in the comments on this PR, but there were too many for me to go looking.

mikecote · 2019-03-04T14:13:46Z

@epixa We had them as a single endpoint before but it served two different purposes. It could either export objects {type, id} or types (string[]) but never a mix of the two. After speaking with @spalger, we decided it would be easier to have two endpoints that make it clear what parameters are required and what to expect in return. Let me know if you see a concern.

epixa · 2019-03-04T16:49:56Z

@mikecote Can you add more details to the REST API docs changes in this PR that describe how you'd use each endpoint and such? It's just not clear to me what the diverging purpose is, but it sounds like I just don't understand what the endpoints do.

elasticmachine · 2019-03-04T18:26:25Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

kobelb

LGTM from the security perspective. Thanks for creating all these tests @mikecote!

…basic-export

spalger

LGTM

elasticmachine · 2019-03-05T20:33:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Initial work for new server side export API * Revert UI changes, API only in this PR * Remove whitespace at top of export.asciidoc * Add tests around limitations * Add comment * Convert some files to typescript * Move Boom.boomify to where the errors are created * Use Boom.badRequest for now * Fix lint issue * Move files * Update tests * Add functional test * Export all documents by default * Update test assertions * Use ~10000 saved objects in export api integration test * Convert route to typescript, add content-type response header * Move some tests to api_integration * Use new sort and rename functions/variables * Move tests to API integration * Cleanup and finalize api integration tests * Make type or objects required but not both in the same call * Add spaces / security tests * Add noTypeOrObjects to security / spaces tests * Use json-stable-stringify and add tests for export ordering * Address self feedback, add without kibana index test * Only allow export API to export index-pattern, dashboard, visualization and search type objects * Make import export size configurable and fix broken tests * Fix broken tests * Move test config to mock server * Add more typescript types instead of using any * Convert request from GET to POST * Fix saved objects mixin test * Update src/legacy/server/saved_objects/lib/export.ts Co-Authored-By: mikecote <[email protected]> * Apply PR feedback * Fix lint error * Update test snapshots due to jest upgrade * Add error handling for bulkGet * Split export API into two endpoints * Update src/legacy/server/saved_objects/routes/export_by_type.test.ts Co-Authored-By: mikecote <[email protected]> * Update docs/api/saved-objects/export_by_type.asciidoc Co-Authored-By: mikecote <[email protected]> * Update docs/api/saved-objects/export_by_type.asciidoc Co-Authored-By: mikecote <[email protected]> * Update src/legacy/server/saved_objects/routes/export_objects.test.ts Co-Authored-By: mikecote <[email protected]> * Apply PR feedback * MockServer -> createMockServer * Revert back to single API * Re-apply PR feedback

rayafratkina · 2019-03-22T16:53:00Z

@mikecote note that in order for release_note:dev_docs label to work properly you need to add a # Dev Docs section to the end of your PR description (see e.g. #28985). That way our automated tool can build the changes blog post from the PRs.

mikecote · 2019-03-22T17:04:16Z

@rayafratkina good catch, that is a very cool feature! I will go through my PRs and fix accordingly.

elasticmachine · 2019-03-27T23:01:31Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

mikecote added 6 commits February 6, 2019 16:58

Initial work for new server side export API

593418e

Revert UI changes, API only in this PR

bd718b4

Remove whitespace at top of export.asciidoc

dcbcfd6

Add tests around limitations

0cb0eb9

Add comment

39f4369

Convert some files to typescript

f686c37

mikecote added WIP Work in progress Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.2.0 labels Feb 6, 2019

mikecote self-assigned this Feb 6, 2019

This comment has been minimized.

Sign in to view

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

47092b9

…basic-export

This comment has been minimized.

Sign in to view

spalger reviewed Feb 7, 2019

View reviewed changes

src/server/saved_objects/routes/export.js Outdated Show resolved Hide resolved

mikecote added 2 commits February 8, 2019 00:00

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

8e62c27

…basic-export

Move Boom.boomify to where the errors are created

05b54a7

mikecote commented Feb 8, 2019

View reviewed changes

src/server/saved_objects/lib/export.ts Outdated Show resolved Hide resolved

Use Boom.badRequest for now

443b16a

This comment has been minimized.

Sign in to view

Fix lint issue

f237623

This comment has been minimized.

Sign in to view

mikecote added the v8.0.0 label Feb 8, 2019

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

41674ff

…basic-export

This comment has been minimized.

Sign in to view

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

a2cc1bb

…basic-export

This comment has been minimized.

Sign in to view

mikecote added 3 commits February 11, 2019 09:30

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

063adcf

…basic-export

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

d071a36

…basic-export

Move files

398e56d

Update src/legacy/server/saved_objects/routes/export_objects.test.ts

c7fc57d

Co-Authored-By: mikecote <[email protected]>

This comment has been minimized.

Sign in to view

Apply PR feedback

00db26d

mikecote requested a review from spalger March 1, 2019 21:34

spalger approved these changes Mar 1, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

Merge with master

10acc3f

This comment has been minimized.

Sign in to view

MockServer -> createMockServer

8198687

kobelb approved these changes Mar 4, 2019

View reviewed changes

mikecote added 3 commits March 5, 2019 14:34

Revert back to single API

89ebd58

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

aa4550a

…basic-export

Re-apply PR feedback

4ae7da1

spalger approved these changes Mar 5, 2019

View reviewed changes

mikecote added the release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. label Mar 5, 2019

mikecote merged commit 399067a into elastic:master Mar 5, 2019

mikecote mentioned this pull request Mar 5, 2019

[7.x] Basic server side export API for saved objects (#30326) #32513

Merged

mikecote added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. and removed release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. labels Mar 25, 2019

mikecote mentioned this pull request Dec 14, 2020

Upgrade to hapi version 20 #85406

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic server side export API for saved objects #30326

Basic server side export API for saved objects #30326

mikecote commented Feb 6, 2019 •

edited

Loading

elasticmachine commented Feb 6, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

spalger left a comment

This comment has been minimized.

epixa commented Mar 4, 2019

mikecote commented Mar 4, 2019

epixa commented Mar 4, 2019

This comment has been minimized.

elasticmachine commented Mar 4, 2019

kobelb left a comment

spalger left a comment

elasticmachine commented Mar 5, 2019

rayafratkina commented Mar 22, 2019

mikecote commented Mar 22, 2019

elasticmachine commented Mar 27, 2019

Basic server side export API for saved objects #30326

Basic server side export API for saved objects #30326

Conversation

mikecote commented Feb 6, 2019 • edited Loading

Dev-Docs

Added server side API for exporting saved objects

elasticmachine commented Feb 6, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

spalger left a comment

Choose a reason for hiding this comment

This comment has been minimized.

epixa commented Mar 4, 2019

mikecote commented Mar 4, 2019

epixa commented Mar 4, 2019

This comment has been minimized.

elasticmachine commented Mar 4, 2019

💚 Build Succeeded

kobelb left a comment

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 5, 2019

💚 Build Succeeded

rayafratkina commented Mar 22, 2019

mikecote commented Mar 22, 2019

elasticmachine commented Mar 27, 2019

💔 Build Failed

mikecote commented Feb 6, 2019 •

edited

Loading