[Fleet] Show pipelines and mappings editor for input packages #154077

hop-dev · 2023-03-30T16:01:57Z

Summary

Show the pipelines and mappings editor for input packages. This can be seen when editing an input package integration policy.

Previously this was hidden for input packages and any integration package which allowed datastream to be configured. I have tweaked this logic so that we check if the matching index template for a stream exists, if it does then show the editor.

This allows us to handle an edge case (which will become more common as input packages are rolled out) when integration package which has since been upgraded to an input package, for example:

user installs the custom logs integration package and sets dataset to helloworld
the logs-helloworld index template is not created as integration packages only create index templates on install (custom logs creates logs-log.log)
there is therefore no ingest pipeline @custom or @packagecomponent template to customize, we do not show the pipeline editor.
The user upgrades to an input package version of custom logs (coming in v2.0.0) there still isn't an index template because we do not create them on upgrade, so we still don't want to show the pipelines and mappings editor
Just checking for input package in this scenario would mean we showed the editor incorrectly

I have opted to show the editors even if the stream is owned by another package e.g custom logs data is being sent to an nginx stream.

Automated tests

I have added cypress tests for checking that the mappings pipelines editors are shown for input packages.

Manual test cases

Here is version 2.0.0 of logs which is an input package: log-2.0.0.zip it can be added to a deployment using curl:

curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/mark/api/fleet/epm/packages -u elastic:changeme --data-binary @/path/to/log-2.0.0.zip

Create a policy for an integration package which does not allow dataset to be configured (e.g Apache)
- pipelines & editor should be visible when editing package policy
Create a policy for an integration package which does allow dataset to be configured (e.g custom logs 1.6.1)
- pipelines & mappings editor should not be shown if dataset doesn't match the one installed by the package logs.log
Upgrade custom logs to the input package version (2.0.0)
- pipelines and mappings editor still should not be shown

apmmachine · 2023-03-30T16:02:23Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-03-30T16:04:52Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2023-03-30T16:49:21Z

...tions/fleet/sections/agent_policy/create_package_policy_page/components/datastream_hooks.tsx


-import { useLink } from '../../../../hooks';
+import { sendRequestForRq, useLink } from '../../../../hooks';


...olicy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

…strea

…-ref HEAD~1..HEAD --fix'

hop-dev · 2023-03-31T13:37:11Z

x-pack/plugins/fleet/cypress/plugins/index.ts

+  }) {
+    const { method, path, body, contentType } = opts;
+    const Authorization = `Basic ${Buffer.from(
+      `elastic:${config.env.ELASTICSEARCH_PASSWORD}`


Does anyone know how to add an env var with the elastic user in? We currently only have one with system_indices_superuser I can't see how they are set!

Have you tried config.env.ELASTICSEARCH_USERNAME ?

kibana-ci · 2023-03-31T17:28:12Z

💚 Build Succeeded

Buildkite Build
Commit: e2fde5b

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	991	993	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	940.8KB	941.4KB	+597.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	125.7KB	126.2KB	+488.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1096	1098	+2

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	512	515	+3

History

💔 Build #117086 failed 061f280

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

criamico · 2023-04-03T09:09:17Z

x-pack/plugins/fleet/cypress/e2e/package_policy_real.cy.ts

+  cy.get('button').contains('Change defaults').click();
+  cy.get('[data-test-subj^="advancedStreamOptionsToggle"]').click();
+}
+describe('Input package create and edit package policy', () => {


Thanks for adding these tests Mark!

criamico · 2023-04-03T09:10:30Z

...ns/fleet/sections/agent_policy/create_package_policy_page/components/datastream_mappings.tsx

+> = ({ packageInputStream, packageInfo, customDataset }) => {
+  const dataStream = customDataset
+    ? { ...packageInputStream.data_stream, dataset: customDataset }
+    : packageInputStream.data_stream;


Maybe you could extract this ternary in a small function to be reused below, since it's the same code?

I think for just 2 uses I'm going to keep it as is as it's not likely to be used anywhere else in the future

juliaElastic · 2023-04-03T09:31:36Z

...tions/fleet/sections/agent_policy/create_package_policy_page/components/datastream_hooks.tsx

+    ['indexTemplateExists', templateName],
+    () =>
+      sendRequestForRq({
+        path: `/api/index_management/index_templates/${templateName}`,


I wonder if this is the best way to call another Kibana API directly or use a service/hook from the index management plugin.

I've just had a go at this but it creates a cyclic dependency in the plugins so I'm going to leave it as it is.

juliaElastic

LGTM

hop-dev added release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team labels Mar 30, 2023

hop-dev self-assigned this Mar 30, 2023

hop-dev marked this pull request as ready for review March 30, 2023 16:04

hop-dev requested a review from a team as a code owner March 30, 2023 16:04

kpollich reviewed Mar 30, 2023

View reviewed changes

hop-dev and others added 8 commits March 31, 2023 14:32

basic viewer working

59ab0f7

only show pipeliens and mappings editor if index template exists for …

8d39052

…strea

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

e225d26

…-ref HEAD~1..HEAD --fix'

test with real package working

0586cec

tests working

e4d81c5

move to getBySel

a44dd61

move to consts

e572506

only enable request when pn edit page

061f280

hop-dev force-pushed the 153105-pipelines-mappings-editor-input-pkg branch from f40bf32 to 061f280 Compare March 31, 2023 13:35

hop-dev requested a review from kpollich March 31, 2023 13:35

hop-dev commented Mar 31, 2023

View reviewed changes

skip snake case check for package zips

e2fde5b

hop-dev requested a review from a team as a code owner March 31, 2023 16:26

jbudz approved these changes Mar 31, 2023

View reviewed changes

juliaElastic requested a review from a team April 3, 2023 09:10

criamico reviewed Apr 3, 2023

View reviewed changes

juliaElastic reviewed Apr 3, 2023

View reviewed changes

juliaElastic approved these changes Apr 3, 2023

View reviewed changes

hop-dev merged commit b80f4fd into elastic:main Apr 3, 2023

hop-dev deleted the 153105-pipelines-mappings-editor-input-pkg branch April 3, 2023 12:05

kibanamachine added the v8.8.0 label Apr 3, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Show pipelines and mappings editor for input packages #154077

[Fleet] Show pipelines and mappings editor for input packages #154077

hop-dev commented Mar 30, 2023 •

edited

Loading

apmmachine commented Mar 30, 2023

elasticmachine commented Mar 30, 2023

kpollich Mar 30, 2023

hop-dev Mar 31, 2023

juliaElastic Apr 3, 2023

kibana-ci commented Mar 31, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

criamico Apr 3, 2023

criamico Apr 3, 2023

hop-dev Apr 3, 2023

juliaElastic Apr 3, 2023

hop-dev Apr 3, 2023

juliaElastic left a comment


		import { useLink } from '../../../../hooks';
		import { sendRequestForRq, useLink } from '../../../../hooks';

[Fleet] Show pipelines and mappings editor for input packages #154077

[Fleet] Show pipelines and mappings editor for input packages #154077

Conversation

hop-dev commented Mar 30, 2023 • edited Loading

Summary

Automated tests

Manual test cases

apmmachine commented Mar 30, 2023

🤖 GitHub comments

elasticmachine commented Mar 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Mar 31, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliaElastic left a comment

Choose a reason for hiding this comment

hop-dev commented Mar 30, 2023 •

edited

Loading