[Fleet] Support granular integrations in policy editor #101531

jen-huang · 2021-06-07T20:49:04Z

Summary

Resolves #93315. This PR completes the second part of integration granularity support (follow up to #99866) by adding support for multiple policy_templates to the integration policy editor.

Screenshots

Add integration page

Edit integration page

Agent YAML with more input blocks

When user navigates from Integration > (any granular integration) > Add (granular integration), only the that integration's inputs are shown in the resulting Add integration page (policy editor). The rest of the package's inputs are hidden.
- The breadcrumbs, title, icon reflect that granular integration
- The cancel actions navigate user back to the granular integration details page
  - TODO: Confirm with product that hiding rest of inputs is ok, if not, b5ba9a4 can be reverted
When user edits such an integration policy (Edit integration page), then all rest of the package's inputs are shown, since we are no longer in the context of a specific granular integration. Instead of inputs being grouped by type like before, each input for each policy template is represented as a collapsible panel.
- The breadcrumbs, title, icon reflect the overall package (no change)
- The cancel actions navigate user back to their prior context (no change)
The resulting agent YAML generated from such policies has more input blocks than before, due to generating an input block for each input for each policy template.
- TODO: Double check with agent team about YAML changes, it should be ok
All changes should be BWC with previous packages (aka packages that only export 1 policy template).

How to test

TLDR Build in-flight AWS package and load it into your local package registry

Get elastic-package https://github.com/elastic/elastic-package
Pull down Adjust aws package to use input groups integrations#767
From that PR, cd into <integrations>packages/aws
Run elastic-package build, make note of the built aws/0.6.1 package at <integrations>/build
Pull down and build package-registry https://github.com/elastic/package-registry
Copy the built 0.6.0 package to <package-registry>/build/package-storage/packages/aws
Run the package registry go run .
Edit kibana.dev.yml to use your local package registry by adding: xpack.fleet.registryUrl: http://localhost:8080
Start Kibana and observe the above changes to policy editor. Play around to make sure I didn't break any other packages.

…ration-policy-editor

…tion, not package

…licy

…ration-policy-editor

jen-huang · 2021-06-08T22:05:09Z

@elasticmachine merge upstream

elasticmachine · 2021-06-08T22:43:19Z

Pinging @elastic/fleet (Team:Fleet)

kpollich

Haven't pulled down to test locally, but code looks great! Had one very minor comment. Awesome work.

kpollich · 2021-06-09T13:28:12Z

x-pack/plugins/fleet/common/services/limited_package.ts

 export const isPackageLimited = (packageInfo: PackageInfo): boolean => {
-  return packageInfo.policy_templates?.[0]?.multiple === false;
+  return !!packageInfo.policy_templates?.find(


Rather than coerce this to a boolean, it might make more sense to use .some to directly return a boolean based on this condition.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/some

…ration-policy-editor

nchaulet · 2021-06-09T19:31:49Z

...plications/fleet/sections/agent_policy/create_package_policy_page/step_configure_package.tsx

+        ? (packageInfo.policy_templates || []).filter(
+            (policyTemplate) => policyTemplate.name === showOnlyIntegration
+          ) || []


nitpick but there is probably a useless || [] here (if it can simplify that expression )

Suggested change

? (packageInfo.policy_templates || []).filter(

(policyTemplate) => policyTemplate.name === showOnlyIntegration

) || []

? (packageInfo.policy_templates || []).filter(

(policyTemplate) => policyTemplate.name === showOnlyIntegration

)

nchaulet

Tested locally adding/editing/removing a few AWS integrations and some non granular integrations and it worked well 🚀

kibanamachine · 2021-06-09T22:21:18Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	985	1006	+21

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	713.7KB	711.3KB	-2.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	425.1KB	433.9KB	+8.8KB

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`ingest-package-policies`	32	33	+1

Unknown metric groups

API count

id	before	after	diff
`fleet`	1075	1096	+21

History

💚 Build #130144 succeeded 60ef874
💚 Build #129965 succeeded 2c9902d29c5f9c40e9567bc5161421239727c96d
💔 Build #129957 failed 681224b
💚 Build #129622 succeeded cb17ec1
💔 Build #129618 failed 5b13f9b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jen-huang

* Adjust packageToPackagePolicy to support multiple policy_templates * Adjust validatePackagePolicy to support multiple policy templates * Create input for every policy template input instead of grouping * Related adjustments for previous commit * Don't key by policy template when package doesn't have integrations * Adjust limited package check to support multiple policy templates * First pass at UI support * Fix imports, add missing schema * Fix finding input agent templates, adjust tests * Only enable specific integrations by default in the UI * Fix import * Make breadcrumbs and create package policy page title reflect integration, not package * Only show specific integration inputs when adding integration to a policy * Change predicate * Simplify condition

kibanamachine · 2021-06-09T22:24:13Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…1848) * Adjust packageToPackagePolicy to support multiple policy_templates * Adjust validatePackagePolicy to support multiple policy templates * Create input for every policy template input instead of grouping * Related adjustments for previous commit * Don't key by policy template when package doesn't have integrations * Adjust limited package check to support multiple policy templates * First pass at UI support * Fix imports, add missing schema * Fix finding input agent templates, adjust tests * Only enable specific integrations by default in the UI * Fix import * Make breadcrumbs and create package policy page title reflect integration, not package * Only show specific integration inputs when adding integration to a policy * Change predicate * Simplify condition Co-authored-by: Jen Huang <[email protected]>

…add-agent-flyout * 'master' of github.com:elastic/kibana: (35 commits) [Cases] Improve connectors mapping (elastic#101145) [ML] Fixes display of job group badges in recognizer wizard (elastic#101775) Fix es_archives path (elastic#101737) [kbnArchiver] convert archive names to root-relative paths (elastic#101839) [Reporting] Make "ScreenCapturePanel" shareable for Canvas (elastic#100623) [Alerting UI] Converted Rules and Connectors management pages to new layout. (elastic#101697) [Fleet] Support granular integrations in policy editor (elastic#101531) [Security Solution][Detections] Update detection alert mappings to ECS v1.10.0 (elastic#101680) [Fleet] Integrations UI: Adjust policies list UI (elastic#101600) chore(NA): moving @kbn/server-route-repository into bazel (elastic#101484) Support owner and description attributes inside the Manifest file, use in API docs (elastic#101786) [Security Solution] fix security empty overview links (elastic#101536) Unskips migration tests now that elastic search is fixed (elastic#101682) Fix endpoint -> integrations onboarding link (elastic#101804) [Alerting] Log warning when rules are not rescheduled due to Saved Object not found error (elastic#101591) Update datafeed_high_count_network_denies.json (elastic#101681) [Index patterns] Field editor example app (elastic#100524) [DOCS] Adding file upload to add data page (elastic#101674) [Security Solution][Endpoint] Adds Endpoint Host Isolation Status common component (elastic#101782) Upgrade ws v7.3.1->v7.4.2 and v6.2.1->v6.2.2 (elastic#101402) ... # Conflicts: # x-pack/plugins/fleet/public/components/agent_enrollment_flyout/agent_policy_selection.tsx # x-pack/plugins/fleet/public/components/agent_enrollment_flyout/index.tsx # x-pack/plugins/fleet/public/components/agent_enrollment_flyout/managed_instructions.tsx # x-pack/plugins/fleet/public/components/agent_enrollment_flyout/standalone_instructions.tsx

…add-integrations-redirect * 'master' of github.com:elastic/kibana: (44 commits) Allow navigating discover flyout via arrow keys (elastic#101772) [Cases] Improve connectors mapping (elastic#101145) [ML] Fixes display of job group badges in recognizer wizard (elastic#101775) Fix es_archives path (elastic#101737) [kbnArchiver] convert archive names to root-relative paths (elastic#101839) [Reporting] Make "ScreenCapturePanel" shareable for Canvas (elastic#100623) [Alerting UI] Converted Rules and Connectors management pages to new layout. (elastic#101697) [Fleet] Support granular integrations in policy editor (elastic#101531) [Security Solution][Detections] Update detection alert mappings to ECS v1.10.0 (elastic#101680) [Fleet] Integrations UI: Adjust policies list UI (elastic#101600) chore(NA): moving @kbn/server-route-repository into bazel (elastic#101484) Support owner and description attributes inside the Manifest file, use in API docs (elastic#101786) [Security Solution] fix security empty overview links (elastic#101536) Unskips migration tests now that elastic search is fixed (elastic#101682) Fix endpoint -> integrations onboarding link (elastic#101804) [Alerting] Log warning when rules are not rescheduled due to Saved Object not found error (elastic#101591) Update datafeed_high_count_network_denies.json (elastic#101681) [Index patterns] Field editor example app (elastic#100524) [DOCS] Adding file upload to add data page (elastic#101674) [Security Solution][Endpoint] Adds Endpoint Host Isolation Status common component (elastic#101782) ... # Conflicts: # x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/index.tsx # x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/details_page/components/package_policies/package_policies_table.tsx

* Adjust packageToPackagePolicy to support multiple policy_templates * Adjust validatePackagePolicy to support multiple policy templates * Create input for every policy template input instead of grouping * Related adjustments for previous commit * Don't key by policy template when package doesn't have integrations * Adjust limited package check to support multiple policy templates * First pass at UI support * Fix imports, add missing schema * Fix finding input agent templates, adjust tests * Only enable specific integrations by default in the UI * Fix import * Make breadcrumbs and create package policy page title reflect integration, not package * Only show specific integration inputs when adding integration to a policy * Change predicate * Simplify condition

jen-huang added 12 commits May 26, 2021 10:20

Adjust packageToPackagePolicy to support multiple policy_templates

58a3650

Adjust validatePackagePolicy to support multiple policy templates

9fa00cb

Create input for every policy template input instead of grouping

86a3c54

Related adjustments for previous commit

a8ee54d

Don't key by policy template when package doesn't have integrations

a055307

Adjust limited package check to support multiple policy templates

dbcd19b

First pass at UI support

ae28351

Merge remote-tracking branch 'upstream/master' into granularity-integ…

1303d89

…ration-policy-editor

Merge remote-tracking branch 'upstream/master' into granularity-integ…

276a1bd

…ration-policy-editor

Fix imports, add missing schema

002420c

Fix finding input agent templates, adjust tests

3cdf6e4

Only enable specific integrations by default in the UI

5b13f9b

jen-huang self-assigned this Jun 7, 2021

jen-huang added Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 v8.0.0 release_note:enhancement auto-backport Deprecated - use backport:version if exact versions are needed labels Jun 7, 2021

Fix import

cb17ec1

jen-huang changed the title ~~[WIP] Support granular integrations in policy editor~~ [WIP][Fleet] Support granular integrations in policy editor Jun 7, 2021

jen-huang added 3 commits June 8, 2021 14:08

Make breadcrumbs and create package policy page title reflect integra…

019ab33

…tion, not package

Only show specific integration inputs when adding integration to a po…

b5ba9a4

…licy

Merge remote-tracking branch 'upstream/master' into granularity-integ…

681224b

…ration-policy-editor

jen-huang changed the title ~~[WIP][Fleet] Support granular integrations in policy editor~~ [Fleet] Support granular integrations in policy editor Jun 8, 2021

jen-huang marked this pull request as ready for review June 8, 2021 22:43

jen-huang requested a review from a team as a code owner June 8, 2021 22:43

kpollich reviewed Jun 9, 2021

View reviewed changes

Change predicate

e108bfb

Merge remote-tracking branch 'upstream/master' into granularity-integ…

60ef874

…ration-policy-editor

jen-huang force-pushed the granularity-integration-policy-editor branch from 2c9902d to 60ef874 Compare June 9, 2021 14:29

nchaulet self-requested a review June 9, 2021 16:49

nchaulet reviewed Jun 9, 2021

View reviewed changes

nchaulet approved these changes Jun 9, 2021

View reviewed changes

Simplify condition

a179cf5

jen-huang enabled auto-merge (squash) June 9, 2021 20:23

jen-huang merged commit 5a5cdb9 into elastic:master Jun 9, 2021

kibanamachine mentioned this pull request Jun 9, 2021

[7.x] [Fleet] Support granular integrations in policy editor (#101531) #101848

Merged

jen-huang deleted the granularity-integration-policy-editor branch June 9, 2021 22:28

jen-huang mentioned this pull request Jun 9, 2021

Meta: enable input groups elastic/package-spec#144

Closed

8 tasks

jen-huang mentioned this pull request Jun 16, 2021

[Fleet] Upgrading a package should trigger a new POLICY CHANGE for agents #92612

Closed

kpollich mentioned this pull request Nov 30, 2021

[Fleet] Fix bug with duplicate Fleet Server inputs in Cloud deployments #119925

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support granular integrations in policy editor #101531

[Fleet] Support granular integrations in policy editor #101531

jen-huang commented Jun 7, 2021 •

edited

Loading

jen-huang commented Jun 8, 2021

elasticmachine commented Jun 8, 2021

kpollich left a comment

kpollich Jun 9, 2021

nchaulet Jun 9, 2021

nchaulet left a comment •

edited

Loading

kibanamachine commented Jun 9, 2021

API count

kibanamachine commented Jun 9, 2021

[Fleet] Support granular integrations in policy editor #101531

[Fleet] Support granular integrations in policy editor #101531

Conversation

jen-huang commented Jun 7, 2021 • edited Loading

Summary

How to test

jen-huang commented Jun 8, 2021

elasticmachine commented Jun 8, 2021

kpollich left a comment

Choose a reason for hiding this comment

kpollich Jun 9, 2021

Choose a reason for hiding this comment

nchaulet Jun 9, 2021

Choose a reason for hiding this comment

nchaulet left a comment • edited Loading

Choose a reason for hiding this comment

kibanamachine commented Jun 9, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

Saved Objects .kibana field count

API count

History

kibanamachine commented Jun 9, 2021

💚 Backport successful

jen-huang commented Jun 7, 2021 •

edited

Loading

nchaulet left a comment •

edited

Loading