[Fleet] Create index templates and ingest pipeline at package policy …

…creation time for input packages (#148772) ## Summary Closes #145529 For integration packages, index templates are created at install time because the package contains all information needed to create the data stream. Input packages need to create the index templates at package policy creation time so that dataset can be populated. Summary of changes: - when creating a package policy for an input package, the correct index templates and ingest pipelines are created, for example for the dataset `dataset1` the following will be created (and added to `installed_es` on the installation saved object): - `logs-dataset1-1.0.0` (ingest_pipeline) - `logs-dataset1` (index_template) - `logs-dataset1@package` (component template) - `logs-dataset1@custom'`(component template) - when a dataset matches an existing data stream - if the existing data stream is from the same package, do not create any new index templates as existing ones will be used - if the existing data stream is from a different package, the API will reject the request unless the force flag is used. - when upgrading an input package, all dynamically created assets will be updated as well. - when uninstalling an input package, all dynamically created assets will be uninstalled - bonus: support the new top level `elasticsearch` field for input package manifests (needed this field for upgrade testing) ### Test setup To test we need a docker registry with input packages, the easiest way is to use the test fixtures from the kibana repo (replace directory with your own) ``` docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` And add this to your kibana yml config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` this will make the test package `input_package_upgrade` available which is a version of the custom logs integration: `http://<your_kibana>/app/integrations/detail/input_package_upgrade-1.0.0/overview` ### Test scenarios #### 1. Package policy creation (new datastream) - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - create a package policy with a valid logfile and `dataset1` as the dataset - logs-dataset1 index template should have been created - add an agent to the package policy - append to the log file - data should be added to the logs-dataset1-default datastream ##### 2. Package policy creation (existing datastream same package) - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - create **another** package policy with a valid logfile and `dataset1` as the dataset - logs-dataset1 should still exist - append to the log file - data should be added to the logs-dataset1-default datastream ##### 3. Package policy creation (existing datastream different package) - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - ensure there are some other fleet data streams on the system (i.e data has been ingested), e.g logs-elastic-agent - create a package policy with a valid logfile and `elastic-agent` as the dataset -the package policy should be successfully created - append to the log file - data should be added to the logs-elastic-agent-default datastream ##### 3b. Package policy creation (existing index template different package) - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - ensure there is another fleet index template on the system with no matching data streams (i.e no data has been ingested), e.g logs-system.auth from the system package - create a package policy with a valid logfile and `system.auth` as the dataset -the package policy should be successfully created - append to the log file - data should be added to the logs-system.auth-default datastream - the `logs-system.auth` index template should still have` _meta.package.name` set to 'system' <img width="650" alt="Screenshot 2023-01-17 at 21 31 10" src="https://user-images.githubusercontent.com/3315046/213016570-daab98e4-9cc2-479a-9349-9fd727f9d899.png"> ##### 4. Package policy delete - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - ensure there are some other fleet data streams on the system, e.g logs-elastic-agent - create one or many package policys with a valid logfile and different datasets - note all of the index templates created - uninstall the package - all created index templates should be deleted ##### 5. package policy upgrade - with input_package_upgrade version 1.0.0 installed and an agent policy with at least one agent - create one or many package policys with a valid logfile and different datasets - note all of the index templates created - upgrade to input_package_upgrade version 1.1.0, this adds `mappings.properties.@timestamp` to the `@package` component template for all data streams: ``` mappings: properties: '@timestamp': ignore_malformed: false type: date ``` - verify all new data streams have the new property ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
elastic · Jan 18, 2023 · f8ecb3b · f8ecb3b
1 parent 7769bb5
commit f8ecb3b
Show file tree

Hide file tree

Showing 45 changed files with 1,446 additions and 239 deletions.
diff --git a/x-pack/plugins/fleet/common/services/policy_template.ts b/x-pack/plugins/fleet/common/services/policy_template.ts
@@ -13,6 +13,7 @@ import type {
   PackageInfo,
   RegistryVarsEntry,
   RegistryDataStream,
+  InstallablePackage,
 } from '../types';
 
 const DATA_STREAM_DATASET_VAR: RegistryVarsEntry = {
@@ -52,7 +53,10 @@ export const getNormalizedInputs = (policyTemplate: RegistryPolicyTemplate): Reg
   return [input];
 };
 
-export const getNormalizedDataStreams = (packageInfo: PackageInfo): RegistryDataStream[] => {
+export const getNormalizedDataStreams = (
+  packageInfo: PackageInfo | InstallablePackage,
+  datasetName?: string
+): RegistryDataStream[] => {
   if (packageInfo.type !== 'input') {
     return packageInfo.data_streams || [];
   }
@@ -66,11 +70,12 @@ export const getNormalizedDataStreams = (packageInfo: PackageInfo): RegistryData
   return policyTemplates.map((policyTemplate) => {
     const dataStream: RegistryDataStream = {
       type: policyTemplate.type,
-      dataset: createDefaultDatasetName(packageInfo, policyTemplate),
+      dataset: datasetName || createDefaultDatasetName(packageInfo, policyTemplate),
       title: policyTemplate.title + ' Dataset',
       release: packageInfo.release || 'ga',
       package: packageInfo.name,
       path: packageInfo.name,
+      elasticsearch: packageInfo.elasticsearch,
       streams: [
         {
           input: policyTemplate.input,
@@ -104,6 +109,6 @@ const addDatasetVarIfNotPresent = (vars?: RegistryVarsEntry[]): RegistryVarsEntr
 };
 
 const createDefaultDatasetName = (
-  packageInfo: PackageInfo,
-  policyTemplate: RegistryPolicyInputOnlyTemplate
+  packageInfo: { name: string },
+  policyTemplate: { name: string }
 ): string => packageInfo.name + '.' + policyTemplate.name;
diff --git a/x-pack/plugins/fleet/common/types/models/package_spec.ts b/x-pack/plugins/fleet/common/types/models/package_spec.ts
@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import type { RegistryPolicyTemplate, RegistryVarsEntry } from './epm';
+import type { RegistryElasticsearch, RegistryPolicyTemplate, RegistryVarsEntry } from './epm';
 
 // Based on https://github.com/elastic/package-spec/blob/master/versions/1/manifest.spec.yml#L8
 export interface PackageSpecManifest {
@@ -27,6 +27,10 @@ export interface PackageSpecManifest {
   policy_templates?: RegistryPolicyTemplate[];
   vars?: RegistryVarsEntry[];
   owner: { github: string };
+  elasticsearch?: Pick<
+    RegistryElasticsearch,
+    'index_template.settings' | 'index_template.mappings'
+  >;
 }
 
 export type PackageSpecPackageType = 'integration' | 'input';

diff --git a/...ons/agent_policy/create_package_policy_page/components/steps/components/dataset_combo.tsx b/...ons/agent_policy/create_package_policy_page/components/steps/components/dataset_combo.tsx
@@ -9,27 +9,52 @@ import React, { useEffect, useState } from 'react';
 import { EuiComboBox } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
 
+import type { DataStream } from '../../../../../../../../../common/types';
+
+interface SelectedDataset {
+  dataset: string;
+  package: string;
+}
+
+const GENERIC_DATASET_NAME = 'generic';
+
 export const DatasetComboBox: React.FC<{
-  value: any;
-  onChange: (newValue: any) => void;
-  datasets: string[];
+  value?: SelectedDataset | string;
+  onChange: (newValue: SelectedDataset) => void;
+  datastreams: DataStream[];
+  pkgName?: string;
   isDisabled?: boolean;
-}> = ({ value, onChange, datasets, isDisabled }) => {
-  const datasetOptions = datasets.map((dataset: string) => ({ label: dataset })) ?? [];
-  const defaultOption = 'generic';
-  const [selectedOptions, setSelectedOptions] = useState<Array<{ label: string }>>([
-    {
-      label: value ?? defaultOption,
-    },
-  ]);
+}> = ({ value, onChange, datastreams, isDisabled, pkgName = '' }) => {
+  const datasetOptions =
+    datastreams.map((datastream: DataStream) => ({
+      label: datastream.dataset,
+      value: datastream,
+    })) ?? [];
+  const existingGenericStream = datasetOptions.find((ds) => ds.label === GENERIC_DATASET_NAME);
+  const valueAsOption = value
+    ? typeof value === 'string'
+      ? { label: value, value: { dataset: value, package: pkgName } }
+      : { label: value.dataset, value: { dataset: value.dataset, package: value.package } }
+    : undefined;
+  const defaultOption = valueAsOption ||
+    existingGenericStream || {
+      label: GENERIC_DATASET_NAME,
+      value: { dataset: GENERIC_DATASET_NAME, package: pkgName },
+    };
+
+  const [selectedOptions, setSelectedOptions] = useState<Array<{ label: string }>>([defaultOption]);
 
   useEffect(() => {
-    if (!value) onChange(defaultOption);
-  }, [value, defaultOption, onChange]);
+    if (!value || typeof value === 'string') onChange(defaultOption.value as SelectedDataset);
+  }, [value, defaultOption.value, onChange, pkgName]);
 
-  const onDatasetChange = (newSelectedOptions: Array<{ label: string }>) => {
+  const onDatasetChange = (newSelectedOptions: Array<{ label: string; value?: DataStream }>) => {
     setSelectedOptions(newSelectedOptions);
-    onChange(newSelectedOptions[0]?.label);
+    const dataStream = newSelectedOptions[0].value;
+    onChange({
+      dataset: newSelectedOptions[0].label,
+      package: !dataStream || typeof dataStream === 'string' ? pkgName : dataStream.package,
+    });
   };
 
   const onCreateOption = (searchValue: string = '') => {
@@ -39,9 +64,13 @@ export const DatasetComboBox: React.FC<{
     }
     const newOption = {
       label: searchValue,
+      value: { dataset: searchValue, package: pkgName },
     };
     setSelectedOptions([newOption]);
-    onChange(searchValue);
+    onChange({
+      dataset: searchValue,
+      package: pkgName,
+    });
   };
   return (
     <EuiComboBox

diff --git a/...gent_policy/create_package_policy_page/components/steps/components/order_datasets.test.ts b/...gent_policy/create_package_policy_page/components/steps/components/order_datasets.test.ts
diff --git a/...ons/agent_policy/create_package_policy_page/components/steps/components/order_datasets.ts b/...ons/agent_policy/create_package_policy_page/components/steps/components/order_datasets.ts
diff --git a/...cy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx b/...cy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx
@@ -42,7 +42,7 @@ import { PackagePolicyEditorDatastreamMappings } from '../../datastream_mappings
 import { ExperimentDatastreamSettings } from './experimental_datastream_settings';
 import { PackagePolicyInputVarField } from './package_policy_input_var_field';
 import { useDataStreamId } from './hooks';
-import { orderDatasets } from './order_datasets';
+import { sortDatastreamsByDataset } from './sort_datastreams';
 
 const ScrollAnchor = styled.div`
   display: none;
@@ -144,9 +144,8 @@ export const PackagePolicyInputStreamConfig = memo<Props>(
     );
 
     const { data: dataStreamsData } = useGetDataStreams();
-    const datasetList =
-      uniq(dataStreamsData?.data_streams.map((dataStream) => dataStream.dataset)) ?? [];
-    const datasets = orderDatasets(datasetList, packageInfo.name);
+    const datasetList = uniq(dataStreamsData?.data_streams) ?? [];
+    const datastreams = sortDatastreamsByDataset(datasetList, packageInfo.name);
 
     return (
       <>
@@ -227,7 +226,8 @@ export const PackagePolicyInputStreamConfig = memo<Props>(
                       errors={inputStreamValidationResults?.vars![varName]}
                       forceShowErrors={forceShowErrors}
                       packageType={packageInfo.type}
-                      datasets={datasets}
+                      packageName={packageInfo.name}
+                      datastreams={datastreams}
                       isEditPage={isEditPage}
                     />
                   </EuiFlexItem>
@@ -289,7 +289,8 @@ export const PackagePolicyInputStreamConfig = memo<Props>(
                             errors={inputStreamValidationResults?.vars![varName]}
                             forceShowErrors={forceShowErrors}
                             packageType={packageInfo.type}
-                            datasets={datasets}
+                            packageName={packageInfo.name}
+                            datastreams={datastreams}
                             isEditPage={isEditPage}
                           />
                         </EuiFlexItem>

diff --git a/...create_package_policy_page/components/steps/components/package_policy_input_var_field.tsx b/...create_package_policy_page/components/steps/components/package_policy_input_var_field.tsx
@@ -22,7 +22,7 @@ import styled from 'styled-components';
 
 import { CodeEditor } from '@kbn/kibana-react-plugin/public';
 
-import type { RegistryVarsEntry } from '../../../../../../types';
+import type { DataStream, RegistryVarsEntry } from '../../../../../../types';
 
 import { MultiTextInput } from './multi_text_input';
 import { DatasetComboBox } from './dataset_combo';
@@ -39,7 +39,8 @@ export const PackagePolicyInputVarField: React.FunctionComponent<{
   forceShowErrors?: boolean;
   frozen?: boolean;
   packageType?: string;
-  datasets?: string[];
+  packageName?: string;
+  datastreams?: DataStream[];
   isEditPage?: boolean;
 }> = memo(
   ({
@@ -50,7 +51,8 @@ export const PackagePolicyInputVarField: React.FunctionComponent<{
     forceShowErrors,
     frozen,
     packageType,
-    datasets = [],
+    packageName,
+    datastreams = [],
     isEditPage = false,
   }) => {
     const [isDirty, setIsDirty] = useState<boolean>(false);
@@ -73,7 +75,8 @@ export const PackagePolicyInputVarField: React.FunctionComponent<{
       if (name === 'data_stream.dataset' && packageType === 'input') {
         return (
           <DatasetComboBox
-            datasets={datasets}
+            pkgName={packageName}
+            datastreams={datastreams}
             value={value}
             onChange={onChange}
             isDisabled={isEditPage}
@@ -168,7 +171,8 @@ export const PackagePolicyInputVarField: React.FunctionComponent<{
       value,
       onChange,
       frozen,
-      datasets,
+      packageName,
+      datastreams,
       isEditPage,
       isInvalid,
       fieldLabel,

diff --git a/...nt_policy/create_package_policy_page/components/steps/components/sort_datastreams.test.ts b/...nt_policy/create_package_policy_page/components/steps/components/sort_datastreams.test.ts
@@ -0,0 +1,33 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { DataStream } from '../../../../../../../../../common/types';
+
+import { sortDatastreamsByDataset } from './sort_datastreams';
+
+const ds = (dataset: string) => ({ dataset } as DataStream);
+describe('orderDatasets', () => {
+  it('should move datasets up that match package name', () => {
+    const datasets = sortDatastreamsByDataset(
+      [ds('system.memory'), ds('elastic_agent'), ds('elastic_agent.filebeat'), ds('system.cpu')],
+      'elastic_agent'
+    );
+
+    expect(datasets).toEqual([
+      ds('elastic_agent'),
+      ds('elastic_agent.filebeat'),
+      ds('system.cpu'),
+      ds('system.memory'),
+    ]);
+  });
+
+  it('should order alphabetically if name does not match', () => {
+    const datasets = sortDatastreamsByDataset([ds('system.memory'), ds('elastic_agent')], 'nginx');
+
+    expect(datasets).toEqual([ds('elastic_agent'), ds('system.memory')]);
+  });
+});
diff --git a/...s/agent_policy/create_package_policy_page/components/steps/components/sort_datastreams.ts b/...s/agent_policy/create_package_policy_page/components/steps/components/sort_datastreams.ts
@@ -0,0 +1,19 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { partition, sortBy } from 'lodash';
+
+import type { DataStream } from '../../../../../../../../../common/types';
+
+// sort data streams by dataset name, but promote datastreams that are from this package to the start
+export function sortDatastreamsByDataset(datasetList: DataStream[], name: string): DataStream[] {
+  const [relevantDatasets, otherDatasets] = partition(sortBy(datasetList, 'dataset'), (record) =>
+    record.dataset.startsWith(name)
+  );
+  const datasets = relevantDatasets.concat(otherDatasets);
+  return datasets;
+}
diff --git a/...cy/create_package_policy_page/multi_page_layout/components/page_steps/add_integration.tsx b/...cy/create_package_policy_page/multi_page_layout/components/page_steps/add_integration.tsx
@@ -28,6 +28,7 @@ import type { PackagePolicyValidationResults } from '../../../services';
 import { validatePackagePolicy, validationHasErrors } from '../../../services';
 import { NotObscuredByBottomBar } from '..';
 import { StepConfigurePackagePolicy, StepDefinePackagePolicy } from '../../../components';
+import { prepareInputPackagePolicyDataset } from '../../../services/prepare_input_pkg_policy_dataset';
 
 const ExpandableAdvancedSettings: React.FC = ({ children }) => {
   const [isShowingAdvanced, setIsShowingAdvanced] = useState<boolean>(false);
@@ -147,7 +148,11 @@ export const AddIntegrationPageStep: React.FC<MultiPageStepLayoutProps> = (props
     force?: boolean;
   }) => {
     setFormState('LOADING');
-    const result = await sendCreatePackagePolicy({ ...newPackagePolicy, force });
+    const { policy, forceCreateNeeded } = await prepareInputPackagePolicyDataset(newPackagePolicy);
+    const result = await sendCreatePackagePolicy({
+      ...policy,
+      force: forceCreateNeeded || force,
+    });
     setFormState('SUBMITTED');
     return result;
   };