add `isExportable` SO export API #101860

pgayvallet · 2021-06-10T08:39:00Z

Summary

Fix #99680

Add a new isExportable property to SavedObjectsTypeManagementDefinition to let type owners have per-object exportability control.
Adapt SO export to use this new feature
Add generic support for SO type attributes when registering a SO type (SavedObjectsType and underlying types)

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…table

pgayvallet · 2021-06-13T19:20:47Z

src/core/server/saved_objects/export/collect_exported_objects.ts

+    // first, evict current objects that are not exportable
+    const {
+      exportable: untransformedExportableInitialObjects,
+      nonExportable: nonExportableInitialObjects,
+    } = await splitByExportability(currentObjects, isExportable);


See #99680 (comment) about the logic of the order.

pgayvallet · 2021-06-13T19:25:22Z

src/core/server/saved_objects/saved_objects_service.ts

-  registerType: (type: SavedObjectsType) => void;
+  registerType: <Attributes = any>(type: SavedObjectsType<Attributes>) => void;


Added generic for the attributes on SavedObjectsType, SavedObjectsTypeManagementDefinition and underlying types. However for BWC, I was forced to default to any instead of unknown for the public API.

How much does this break if we switch to unknown? I'm guessing a lot 😓

Yea, basically all types registering a getUrl or getTitle management handler, which is a lot. This is why I don't think we should do it in this PR. I can open a follow-up and ping the teams, though

pgayvallet · 2021-06-13T19:29:42Z

x-pack/test/saved_object_api_integration/common/suites/export.ts

+const EMPTY_RESULT = {
+  excludedObjects: [],
+  excludedObjectsCount: 0,


Took me a while to understand why adding the new properties at the end of the definition was still causing a failure. TIL, for some test suites, we're asserting against the stringified so the properties needs to be in the exact same order.

🤔 @jportner do recall why we are doing a stringified test here? Can we make this more robust by sorting the keys before stringifying, and switch to using json-stable-stringify instead?

IIRC it's because the export API returns Content-Type: application/ndjson and the current approach was cleaner. Otherwise we would have to do this

diff --git a/x-pack/test/saved_object_api_integration/common/suites/export.ts b/x-pack/test/saved_object_api_integration/common/suites/export.ts index d9ebbac8102..ea474a88092 100644 --- a/x-pack/test/saved_object_api_integration/common/suites/export.ts +++ b/x-pack/test/saved_object_api_integration/common/suites/export.ts @@ -135,8 +135,6 @@ export const createRequest = ({ type, id }: ExportTestCase) => const getTestTitle = ({ failure, title }: ExportTestCase) => `${failure?.reason || 'success'} ["${title}"]`; -const EMPTY_RESULT = { exportedCount: 0, missingRefCount: 0, missingReferences: [] }; - export function exportTestSuiteFactory(esArchiver: any, supertest: SuperTest<any>) { const expectSavedObjectForbiddenBulkGet = expectResponses.forbiddenTypes('bulk_get'); const expectResponseBody = (testCase: ExportTestCase): ExpectResponseBody => async ( @@ -151,7 +149,12 @@ export function exportTestSuiteFactory(esArchiver: any, supertest: SuperTest<any } else if (failure.statusCode === 200) { // "find" was unauthorized, which returns an empty result expect(response.body).not.to.have.property('error'); - expect(response.text).to.equal(JSON.stringify(EMPTY_RESULT)); + const exportDetails = JSON.parse(response.text); + expect(exportDetails).to.eql({ + exportedCount: 0, + missingRefCount: 0, + missingReferences: [], + }); } else { throw new Error(`Unexpected failure status code: ${failure.statusCode}`); }

The success case assertions further down also have to parse the ndjson.

elasticmachine · 2021-06-13T19:49:06Z

Pinging @elastic/kibana-core (Team:Core)

joshdover

Do we have a known use case for async isExportable implementations? If not, this could be something we opt not to support at this time to avoid any performance issues with this new hook.

joshdover · 2021-06-15T12:11:45Z

src/core/server/saved_objects/saved_objects_service.ts

-  registerType: (type: SavedObjectsType) => void;
+  registerType: <Attributes = any>(type: SavedObjectsType<Attributes>) => void;


How much does this break if we switch to unknown? I'm guessing a lot 😓

src/core/server/saved_objects/types.ts

src/core/server/saved_objects/export/collect_exported_objects.ts

joshdover · 2021-06-15T12:32:01Z

src/core/server/saved_objects/export/saved_objects_exporter.test.ts

-                },
-              ]
-          `);
+        Array [


Ugh all these snapshots shakes fist

src/plugins/saved_objects_management/public/lib/extract_export_details.test.ts

...ins/saved_objects_management/public/management_section/objects_table/saved_objects_table.tsx

legrego

A few nits below, and a general comment:

I want to make sure that we don't use this new isExportable function to enforce authorization/access control. We currently grant users with the Saved Objects Management feature access to any exportable saved object type, so if we start relying on isExportable to refine that authorization, then we will have a mismatch of expectations. It doesn't sound like that's the intent here, but I wanted to double check.

src/core/server/saved_objects/export/collect_exported_objects.ts

legrego · 2021-06-15T12:25:27Z

src/core/server/saved_objects/export/collect_exported_objects.ts

+  const nonExportableObjects: SavedObject[] = [];
+  await Promise.all(
+    objects.map(async (obj) => {
+      const exportable = await isExportable(obj);


If saved object migrations have taught me anything, it's that plugin authors can/will create buggy functions to interrogate their saved objects. Can we be defensive here, and handle any thrown errors explicitly? I'm open to suggestions, but I'm thinking we could either:

fail open, and treat failed checks as "exportable"

fail closed, and treat failed checks as "unexportable"

raise our own error in response to a failed check

Option 3 is my least favorite, because it prevents administrators from exporting saved objects due to a bug in our code, without any recourse. That said, I don't know the intent of preventing objects from being excluded, so I don't know if it's safe for us to always fail open/closed.

Perhaps the safest option is to "fail closed", and report the failure within the export metadata and/or in a server log entry. What do you think?

If saved object migrations have taught me anything, it's that plugin authors can/will create buggy functions to interrogate their saved objects

Can't disagree on that

Perhaps the safest option is to "fail closed", and report the failure within the export metadata and/or in a server log entry

That's a perfect use case for the currently unused SavedObjectsExportExcludedObject.reason

src/core/server/saved_objects/export/collect_exported_objects.ts

legrego · 2021-06-15T12:33:49Z

x-pack/test/saved_object_api_integration/common/suites/export.ts

+const EMPTY_RESULT = {
+  excludedObjects: [],
+  excludedObjectsCount: 0,


🤔 @jportner do recall why we are doing a stringified test here? Can we make this more robust by sorting the keys before stringifying, and switch to using json-stable-stringify instead?

pgayvallet · 2021-06-15T13:15:54Z

@legrego

We currently grant users with the Saved Objects Management feature access to any exportable saved object type, so if we start relying on isExportable to refine that authorization, then we will have a mismatch of expectations. It doesn't sound like that's the intent here, but I wanted to double check.

I confirm that it is not the intent at all. This is only required for the '8.0 exportability' project, and AFAIK it may even be only temporary. See #99680 and #50266 (comment) for more context.

…table

pgayvallet · 2021-06-16T07:13:35Z

@joshdover @legrego I think I addressed all of your feedbacks. PTAL

src/core/server/saved_objects/export/collect_exported_objects.ts

test/plugin_functional/test_suites/saved_objects_management/export_transform.ts

…table

legrego

LGTM, thanks for the edits!

joshdover

LGTM

…table

kibanamachine · 2021-06-21T07:56:12Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`core`	1053	1054	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`savedObjectsManagement`	138.5KB	138.9KB	+430.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`core`	26	27	+1

Unknown metric groups

API count

id	before	after	diff
`core`	2276	2283	+7

History

💚 Build #132399 succeeded 17dd0d1
💚 Build #132025 succeeded 7fef467
💚 Build #131570 succeeded 5b0ff7e
💔 Build #131503 failed a8ad0b6
💔 Build #131474 failed 55f03c2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* add `isExportable` SO export API * add warning when export contains excluded objects * add FTR test * fix API integration assertions * lint * fix assertions again * doc * update generated doc * fix esarchiver paths * use maps instead of objects * SavedObjectsExportablePredicate is no longer async * more docs * generated doc * use info instead of warning when export contains excluded objects * try/catch on isExportable call and add exclusion reason * add FTR test for errored objects * log error if isExportable throws

* add `isExportable` SO export API (#101860) * add `isExportable` SO export API * add warning when export contains excluded objects * add FTR test * fix API integration assertions * lint * fix assertions again * doc * update generated doc * fix esarchiver paths * use maps instead of objects * SavedObjectsExportablePredicate is no longer async * more docs * generated doc * use info instead of warning when export contains excluded objects * try/catch on isExportable call and add exclusion reason * add FTR test for errored objects * log error if isExportable throws * fix dataset for 7.x

pgayvallet added 6 commits June 10, 2021 10:38

add isExportable SO export API

434d650

add warning when export contains excluded objects

8ddccee

add FTR test

d5762d6

fix API integration assertions

b2a92fe

lint

9c4e1d2

fix assertions again

0e269f5

pgayvallet added release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.14.0 labels Jun 10, 2021

pgayvallet added 3 commits June 10, 2021 16:39

doc

1ea546b

update generated doc

6a8b06a

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

49af21e

…table

pgayvallet commented Jun 13, 2021

View reviewed changes

pgayvallet marked this pull request as ready for review June 13, 2021 19:49

pgayvallet requested review from a team as code owners June 13, 2021 19:49

fix esarchiver paths

d2a3d98

joshdover reviewed Jun 15, 2021

View reviewed changes

legrego reviewed Jun 15, 2021

View reviewed changes

pgayvallet added 9 commits June 15, 2021 15:20

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

78e63ac

…table

use maps instead of objects

8bd90b4

SavedObjectsExportablePredicate is no longer async

6e85351

more docs

08cd885

generated doc

16ac169

use info instead of warning when export contains excluded objects

55f03c2

try/catch on isExportable call and add exclusion reason

a8ad0b6

add FTR test for errored objects

95ffef9

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

5b0ff7e

…table

pgayvallet requested review from joshdover and legrego June 16, 2021 07:13

legrego reviewed Jun 16, 2021

View reviewed changes

src/core/server/saved_objects/export/collect_exported_objects.ts Show resolved Hide resolved

test/plugin_functional/test_suites/saved_objects_management/export_transform.ts Outdated Show resolved Hide resolved

pgayvallet added 2 commits June 17, 2021 13:10

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

2a9330f

…table

log error if isExportable throws

7fef467

legrego approved these changes Jun 17, 2021

View reviewed changes

joshdover approved these changes Jun 18, 2021

View reviewed changes

pgayvallet added 2 commits June 18, 2021 15:25

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

17dd0d1

…table

Merge remote-tracking branch 'upstream/master' into kbn-99680-isExpor…

b49ef4f

…table

pgayvallet merged commit 59d7f33 into elastic:master Jun 21, 2021

pgayvallet mentioned this pull request Jun 21, 2021

[7.x] add isExportable SO export API (#101860) #102698

Merged

ymao1 mentioned this pull request Jun 23, 2021

[Alerting] Enable rule import/export and allow rule types to exclude themselves from export #102999

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add `isExportable` SO export API #101860

add `isExportable` SO export API #101860

pgayvallet commented Jun 10, 2021 •

edited

Loading

pgayvallet Jun 13, 2021

pgayvallet Jun 13, 2021

joshdover Jun 15, 2021

pgayvallet Jun 15, 2021

pgayvallet Jun 13, 2021

legrego Jun 15, 2021

jportner Jun 15, 2021

elasticmachine commented Jun 13, 2021

joshdover left a comment

joshdover Jun 15, 2021

joshdover Jun 15, 2021

legrego left a comment

legrego Jun 15, 2021

pgayvallet Jun 15, 2021

legrego Jun 15, 2021

pgayvallet commented Jun 15, 2021

pgayvallet commented Jun 16, 2021

legrego left a comment

joshdover left a comment

kibanamachine commented Jun 21, 2021

API count

		registerType: (type: SavedObjectsType) => void;
		registerType: <Attributes = any>(type: SavedObjectsType<Attributes>) => void;

add isExportable SO export API #101860

add isExportable SO export API #101860

Conversation

pgayvallet commented Jun 10, 2021 • edited Loading

Summary

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 13, 2021

joshdover left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legrego left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet commented Jun 15, 2021

pgayvallet commented Jun 16, 2021

legrego left a comment

Choose a reason for hiding this comment

joshdover left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 21, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Public APIs missing exports

API count

History

add `isExportable` SO export API #101860

add `isExportable` SO export API #101860

pgayvallet commented Jun 10, 2021 •

edited

Loading