Basic server side import API for saved objects #32158

mikecote · 2019-02-27T21:27:34Z

PR implements the first steps of the saved objects import API for #4759.

Two basic endpoint are added under POST /api/saved_objects/_import and POST /api/saved_objects/_resolve_import_conflicts that throws an error after 10,000 objects. The removal of the limit will come in a future PR when using streams to optimize scalability and performance.

Also a future PR will change the management UI.

Dev-Docs

Added server side API for importing saved objects

New endpoints /api/saved_objects/_import and /api/saved_objects/_resolve_import_errors now exists to import saved objects.

elasticmachine · 2019-02-27T21:27:36Z

Pinging @elastic/kibana-platform

src/legacy/utils/streams/index.d.ts

…basic-import

spalger

Left the security api suites for @kobelb to review, but looking great so far!

docs/api/saved-objects/import.asciidoc

docs/api/saved-objects/resolve_import_conflicts.asciidoc

src/legacy/server/saved_objects/lib/import.ts

spalger · 2019-03-06T22:36:11Z

src/legacy/utils/streams/limit_stream.ts

+    objectMode: true,
+    async transform(obj, enc, done) {
+      if (counter >= limit) {
+        return done(new Error(`Limit of ${limit} objects reached`));


We should probably verify that the import API behaves appropriately when more than 10000 objects are imported, since we're using createConcatStream() we don't have to worry about partial imports happening, but I'm pretty sure it will respond with a 500 and no meaningful error in this case, when ideally we should be responding with a 4xx response that instructs the user to send less objects. Maybe we want to be able to customize the behavior of createLimitStream() when the limit is encountered? Or maybe we should just define this Transform stream inline so we have full control of the behavior in import.ts?

@spalger Interesting, I did the test and yeah it does throw a 500. Because of this, I moved createLimitStream inline to throw a 400 error. I've added functional tests for this as well.

As I changed this, I discovered the API doesn't accept files larger than 1MB. I had to add a config for what size we want to allow on import. I set the default to 50MB? The 10k object es archive is about 20MB when exported from the API.

Let me know your thoughts about these changes.

Sounds good to me, but I wonder what Kibana performance is like when a 20 MB import is loaded into memory... maybe we should consider lowering the default limit to 5000 or something

As discussed and after doing some research, the import for 10k records takes about 3s and blocks the event loop up to ~800ms due to garbage collection. The fix will come when we use streams to not have all the objects in memory. We will leave the limit as is (limit is also configurable by users).

test/api_integration/apis/saved_objects/import.js

test/api_integration/apis/saved_objects/resolve_import_conflicts.js

…basic-import

src/legacy/server/saved_objects/lib/import.ts

src/legacy/server/config/schema.js

kobelb · 2019-03-07T21:51:08Z

x-pack/test/saved_object_api_integration/common/suites/import.ts

+    const { user = {}, spaceId = DEFAULT_SPACE_ID, tests } = definition;
+
+    describeFn(description, () => {
+      before(() => esArchiver.load('saved_objects/spaces'));


Can we add a test case for when we try to import a file which we're only authorized to import some of the saved object types? We should be able to accomplish this by fabricating a saved object type, similar to what we do with the find APIs.

I've added tests to cover this scenario, we can iterate on the approach when you get a chance.

x-pack/test/saved_object_api_integration/common/suites/resolve_import_conflicts.ts

…basic-import

elasticmachine · 2019-03-08T05:24:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger

LGTM

* Initial work * Add overwrite and skip support * Cleanup and add tests * Move code into separate files * Remove reduce * New API parameters * Add support to replace references * Add better error handling * Add spaces tests * Fix return type in collectSavedObjects * Apply PR feedback * Update jest tests due to jest version upgrade * Add docs * WIP * Split import routes pt1 * Add tests * Fix broken tests * Update docs and fix broken test * Add successCount to _import endpoint * Make skip by default in resolution API * Update tests for removal of skips * Add back support for skips * Add success count * Add back resolve import conflicts x-pack tests * Remove writev from filter stream * Delete _mock_server.d.ts file * Rename lib/import_saved_objects to lib/import * Filter records at stream level for conflict resolution * Update docs * Add tests to validate documentation * Return 200 instead of other code for errors, include errors array * Change [] to {} * Apply PR feedback * Fix import object limit to not return 500 * Change some wording in the docs * Fix status code * Apply PR feedback pt2 * Lower maxImportPayloadBytes to 10MB * Add unknown type tests for import * Add unknown type tests for resolve_import_conflicts * Fix tslint issues

spalger · 2019-03-08T19:57:01Z

💃🎉💃🎉💃🎉💃🎉💃🎉💃🎉💃

elasticmachine · 2019-03-27T23:00:49Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

mikecote added 9 commits February 27, 2019 16:21

Initial work

c2512a4

Add overwrite and skip support

720699b

Cleanup and add tests

2308bc5

Move code into separate files

4386021

Remove reduce

20a1329

New API parameters

8dfb656

Add support to replace references

d04ceb9

Add better error handling

c6d3dc1

Add spaces tests

96233a8

mikecote added release_note:enhancement Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 v7.2.0 labels Feb 27, 2019

mikecote self-assigned this Feb 27, 2019

spalger reviewed Feb 27, 2019

View reviewed changes

src/legacy/utils/streams/index.d.ts Outdated Show resolved Hide resolved

src/legacy/utils/streams/index.d.ts Outdated Show resolved Hide resolved

Fix return type in collectSavedObjects

82f7090

mikecote mentioned this pull request Feb 28, 2019

Basic server side export API for saved objects #30326

Merged

mikecote added 10 commits February 28, 2019 10:10

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

25d2065

…basic-import

Apply PR feedback

3c3623f

Update jest tests due to jest version upgrade

e4f7a41

Add docs

ee87cbb

WIP

324f6c1

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

3043d41

…basic-import

Split import routes pt1

b51ec73

Merge

ab1797a

Add tests

b70d7de

Fix broken tests

81b7fa9

This comment has been minimized.

Sign in to view

Update docs and fix broken test

dc94039

mikecote requested a review from a team March 6, 2019 21:20

mikecote added the review label Mar 6, 2019

spalger reviewed Mar 6, 2019

View reviewed changes

mikecote added 5 commits March 7, 2019 09:01

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

e3047f1

…basic-import

Apply PR feedback

61abed5

Fix import object limit to not return 500

5f0c0b6

Change some wording in the docs

ebd39d9

Fix status code

72c421d

mikecote requested a review from spalger March 7, 2019 16:04

This comment has been minimized.

Sign in to view

spalger reviewed Mar 7, 2019

View reviewed changes

src/legacy/server/saved_objects/lib/import.ts Outdated Show resolved Hide resolved

src/legacy/server/saved_objects/lib/import.ts Outdated Show resolved Hide resolved

Apply PR feedback pt2

7cd2b50

mikecote requested a review from spalger March 7, 2019 20:51

This comment has been minimized.

Sign in to view

kobelb reviewed Mar 7, 2019

View reviewed changes

mikecote added 5 commits March 7, 2019 22:13

Merge branch 'master' of github.com:elastic/kibana into saved-object/…

6f51a75

…basic-import

Lower maxImportPayloadBytes to 10MB

a511dca

Add unknown type tests for import

0f8c3ae

Add unknown type tests for resolve_import_conflicts

d7f8956

Fix tslint issues

162c595

kobelb approved these changes Mar 8, 2019

View reviewed changes

spalger approved these changes Mar 8, 2019

View reviewed changes

mikecote merged commit 7cf9131 into elastic:master Mar 8, 2019

mikecote mentioned this pull request Mar 8, 2019

[7.x] Basic server side import API for saved objects (#32158) #32790

Merged

mikecote added the release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. label Mar 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic server side import API for saved objects #32158

Basic server side import API for saved objects #32158

mikecote commented Feb 27, 2019 •

edited

Loading

elasticmachine commented Feb 27, 2019

This comment has been minimized.

spalger left a comment

spalger Mar 6, 2019 •

edited

Loading

mikecote Mar 7, 2019

spalger Mar 7, 2019 •

edited

Loading

mikecote Mar 7, 2019

This comment has been minimized.

This comment has been minimized.

kobelb Mar 7, 2019

mikecote Mar 8, 2019

elasticmachine commented Mar 8, 2019

spalger left a comment

spalger commented Mar 8, 2019

elasticmachine commented Mar 27, 2019

Basic server side import API for saved objects #32158

Basic server side import API for saved objects #32158

Conversation

mikecote commented Feb 27, 2019 • edited Loading

Dev-Docs

Added server side API for importing saved objects

elasticmachine commented Feb 27, 2019

This comment has been minimized.

spalger left a comment

Choose a reason for hiding this comment

spalger Mar 6, 2019 • edited Loading

Choose a reason for hiding this comment

mikecote Mar 7, 2019

Choose a reason for hiding this comment

spalger Mar 7, 2019 • edited Loading

Choose a reason for hiding this comment

mikecote Mar 7, 2019

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

kobelb Mar 7, 2019

Choose a reason for hiding this comment

mikecote Mar 8, 2019

Choose a reason for hiding this comment

elasticmachine commented Mar 8, 2019

💚 Build Succeeded

spalger left a comment

Choose a reason for hiding this comment

spalger commented Mar 8, 2019

elasticmachine commented Mar 27, 2019

💔 Build Failed

mikecote commented Feb 27, 2019 •

edited

Loading

spalger Mar 6, 2019 •

edited

Loading

spalger Mar 7, 2019 •

edited

Loading