From 7c3887309cec54cc21e1abf8a2522afa49147712 Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 15 Oct 2024 22:51:25 -0300 Subject: [PATCH] [Security Solution] Extend upgrade perform endpoint logic (#191439) Fixes: https://github.com/elastic/kibana/issues/166376 (main ticket) Fixes: https://github.com/elastic/kibana/issues/186544 (handling of specific fields) Fixes: https://github.com/elastic/kibana/issues/180195 (replace PATCH with PUT logic on rule upgrade) ## Summary - Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way that works with prebuilt rules customized by users and resolve conflicts between user customizations and updates from Elastic. - Handles special fields under the hood (see below) - Replaces the update prebuilt rule logic to work with PUT instead of PATCH. ### Rough implementation plan - For each `upgradeableRule`, we attempt to build the payload necessary to pass to `upgradePrebuiltRules()`, which is of type `PrebuiltRuleAsset`. So we retrieve the field names from `FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them. - If any of those `field`s are non-upgreadable, (i.e. its value needs to be handled under the hood) we do so in `determineFieldUpgradeStatus`. - Otherwise, we continue to build a `FieldUpgradeSpecifier` for each field, which will help us determine if that field needs to be set to the base, current, target version, OR if it needs to be calculated as a MERGED value, or it is passed in the request payload as a RESOLVED value. - Notice that we are iterating over "flat" (non-grouped) fields which are part of the `PrebuiltRuleAsset` schema. This means that mapping is necessary between these flat fields and the diffable (grouped) fields that are used in the API contract, part of `DiffableRule`. For example, if we try to determine the value for the `query` field, we will need to look up for its value in the `eql_query` field if the target rule is `eql` or in `esql_query` if the target rule is `esql`. All these mappings can be found in `diffable_rule_fields_mappings.ts`. - Once a `FieldUpgradeSpecifier` has been retrieved for each field of the payload we are building, retrieve its actual value: either fetching it from the base, current or target versions of the rule, from the three way diff calculation, or retrieving it from the request payload if it resolved. - Do this for all upgreadable rules, and the pass the payload array into `upgradePrebuiltRules()`. - **IMPORTANT:** The upgrade prebuilt rules logic has been changed from PATCH to PUT. That means that if the next version of a rule removes a field, and the user updates to that target version, those fields will be undefined in the resulting rule. **Additional example:** a installs a rule, and creates a `timeline_id` for it rule by modifying it. If neither the next version (target version) still does not have a `timeline_id` field for it, and the user updates to that target version fully (without resolving the conflict), that field will not exist anymore in the resulting rule. ## Acceptance criteria - [x] Extend the contract of the API endpoint according to the [POC](https://github.com/elastic/kibana/pull/144060): - [x] Add the ability to pick the `MERGED` version for rule upgrades. If the `MERGED` version is selected, the diffs are recalculated and the rule fields are updated to the result of the diff calculation. This is only possible if all field diffs return a `conflict` value of either `NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`, reject the request with an error specifying that there are conflicts, and that they must be resolved on a per-field basis. - [x] Calculate diffs inside this endpoint, when the value of `pick_version` is `MERGED`. - [x] Add the ability to specify rule field versions, to update specific fields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' | 'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in [PoC](https://github.com/elastic/kibana/pull/144060) for details) ## Handling of special fields Specific fields are handled under the hood based on https://github.com/elastic/kibana/issues/186544 See implementation in `x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`, which imports fields to handle under the hood: - `DiffableFieldsToOmit` - `FieldsToUpdateToCurrentVersion` ## Edge cases - [x] If target version of rule has a **rule type change**, check that all `pick_version`, at all levels, match `TARGET`. Otherwise, create new error and add to ruleErrors array. - [x] if a rule has a specific `targetVersion.type` (for example, EQL) and the user includes in its `fields` object of the request payload any fields which do not match that rule type (in this case, for example, sending in `machine_learning_job_id` as part of `fields`), throw an error for that rule. - [x] Calculation of field diffs: what happens if some fields have a conflict value of `NON_SOLVABLE`: - [x] If the whole rule is being updated to `MERGED`, and **ANY** fields return with a `NON_SOLVABLE` conflict, reject the whole update for that rule: create new error and add to ruleErrors array. - [x] **EXCEPTION** for case above: the whole rule is being updated to `MERGED`, and one or more of the fields return with a `NON_SOLVABLE` conflict, BUT those same fields have a specific `pick_version` for them in the `fields` object which **ARE NOT** `MERGED`. No error should be reported in this case. - [x] The whole rule is being updated to any `pick_version` other than MERGED, but any specific field in the `fields` object is set to upgrade to `MERGED`, and the diff for that fields returns a `NON_SOLVABLE` conflict. In that case, create new error and add to ruleErrors array. ### TODO - [[Security Solution] Add InvestigationFields and AlertSuppression fields to the upgrade workflow [#190597]](https://github.com/elastic/kibana/issues/190597): InvestigationFields is already working, but AlertSuppression is still currently handled under the hood to update to current version. ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Maxim Palenov --- .../model/diff/diffable_rule/diffable_rule.ts | 42 +- .../perform_rule_upgrade_route.ts | 51 +- .../get_prebuilt_rules_status_route.ts | 4 +- .../perform_rule_installation_route.ts | 4 +- ...rt_diffable_fields_match_rule_type.test.ts | 60 ++ .../assert_diffable_fields_match_rule_type.ts | 40 + .../assert_pick_version_is_target.test.ts | 131 +++ .../assert_pick_version_is_target.ts | 48 + .../create_field_upgrade_specifier.test.ts | 118 +++ .../create_field_upgrade_specifier.ts | 72 ++ .../create_props_to_rule_type_map.ts | 43 + .../create_upgradeable_rules_payload.ts | 145 +++ .../diffable_rule_fields_mappings.ts | 211 +++++ .../get_field_predefined_value.test.ts | 65 ++ .../get_field_predefined_value.ts | 73 ++ .../get_upgradeable_rules.test.ts | 191 ++++ .../get_upgradeable_rules.ts | 83 ++ .../get_value_for_field.ts | 94 ++ .../get_value_from_rule_version.ts | 94 ++ .../perform_rule_upgrade_route.ts | 122 +-- .../review_rule_installation_route.ts | 4 +- .../review_rule_upgrade_route.ts | 4 +- .../prebuilt_rule_assets_client.ts | 2 +- .../fetch_rule_versions_triad.ts | 2 +- .../rule_versions/rule_version_specifier.ts | 0 .../rule_assets/prebuilt_rule_asset.mock.ts | 200 +++- .../model/rule_assets/prebuilt_rule_asset.ts | 30 +- .../get_rule_groups.ts} | 34 +- .../mergers/apply_rule_patch.ts | 2 + .../methods/upgrade_prebuilt_rule.ts | 14 +- .../update_actions.ts | 14 + .../get_prebuilt_rules_status.ts | 16 +- .../trial_license_complete_tier/index.ts | 2 + ...e_perform_prebuilt_rules.all_rules_mode.ts | 490 ++++++++++ ...form_prebuilt_rules.specific_rules_mode.ts | 861 ++++++++++++++++++ .../upgrade_prebuilt_rules.ts | 25 +- ...prebuilt_rules_with_historical_versions.ts | 12 +- .../update_prebuilt_rules_package.ts | 15 +- .../export_rules.ts | 1 + .../get_custom_query_rule_params.ts | 1 + .../create_prebuilt_rule_saved_objects.ts | 29 +- .../utils/rules/prebuilt_rules/index.ts | 2 +- ...s.ts => perform_upgrade_prebuilt_rules.ts} | 19 +- 43 files changed, 3202 insertions(+), 268 deletions(-) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_props_to_rule_type_map.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_upgradeable_rules_payload.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/diffable_rule_fields_mappings.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_for_field.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_from_rule_version.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/{model => logic}/rule_versions/rule_version_specifier.ts (100%) rename x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/{rule_versions/get_version_buckets.ts => rule_groups/get_rule_groups.ts} (75%) create mode 100644 x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.all_rules_mode.ts create mode 100644 x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.specific_rules_mode.ts rename x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/{upgrade_prebuilt_rules.ts => perform_upgrade_prebuilt_rules.ts} (67%) diff --git a/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts b/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts index d0a4aa12533e0..6e24b902995f4 100644 --- a/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts +++ b/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts @@ -174,6 +174,17 @@ export const DiffableNewTermsFields = z.object({ alert_suppression: AlertSuppression.optional(), }); +export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [ + DiffableCustomQueryFields, + DiffableSavedQueryFields, + DiffableEqlFields, + DiffableEsqlFields, + DiffableThreatMatchFields, + DiffableThresholdFields, + DiffableMachineLearningFields, + DiffableNewTermsFields, +]); + /** * Represents a normalized rule object that is suitable for passing to the diff algorithm. * Every top-level field of a diffable rule can be compared separately on its own. @@ -200,18 +211,6 @@ export const DiffableNewTermsFields = z.object({ * NOTE: Every top-level field in a DiffableRule MUST BE LOGICALLY INDEPENDENT from other * top-level fields. */ - -export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [ - DiffableCustomQueryFields, - DiffableSavedQueryFields, - DiffableEqlFields, - DiffableEsqlFields, - DiffableThreatMatchFields, - DiffableThresholdFields, - DiffableMachineLearningFields, - DiffableNewTermsFields, -]); - export type DiffableRule = z.infer; export const DiffableRule = z.intersection(DiffableCommonFields, DiffableFieldsByTypeUnion); @@ -246,3 +245,22 @@ export const DiffableAllFields = DiffableCommonFields.merge( .merge(DiffableMachineLearningFields.omit({ type: true })) .merge(DiffableNewTermsFields.omit({ type: true })) .merge(z.object({ type: DiffableRuleTypes })); + +const getRuleTypeFields = (schema: z.ZodObject): string[] => + Object.keys(schema.shape); + +const createDiffableFieldsPerRuleType = (specificFields: z.ZodObject): string[] => [ + ...getRuleTypeFields(DiffableCommonFields), + ...getRuleTypeFields(specificFields), +]; + +export const DIFFABLE_RULE_TYPE_FIELDS_MAP = new Map([ + ['query', createDiffableFieldsPerRuleType(DiffableCustomQueryFields)], + ['saved_query', createDiffableFieldsPerRuleType(DiffableSavedQueryFields)], + ['eql', createDiffableFieldsPerRuleType(DiffableEqlFields)], + ['esql', createDiffableFieldsPerRuleType(DiffableEsqlFields)], + ['threat_match', createDiffableFieldsPerRuleType(DiffableThreatMatchFields)], + ['threshold', createDiffableFieldsPerRuleType(DiffableThresholdFields)], + ['machine_learning', createDiffableFieldsPerRuleType(DiffableMachineLearningFields)], + ['new_terms', createDiffableFieldsPerRuleType(DiffableNewTermsFields)], +]); diff --git a/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts b/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts index c7d3227ef03f3..784f75d09bd7a 100644 --- a/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts +++ b/x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts @@ -11,11 +11,52 @@ import { RuleResponse } from '../../model/rule_schema/rule_schemas.gen'; import { AggregatedPrebuiltRuleError, DiffableAllFields } from '../model'; import { RuleSignatureId, RuleVersion } from '../../model'; +export type Mode = z.infer; +export const Mode = z.enum(['ALL_RULES', 'SPECIFIC_RULES']); +export type ModeEnum = typeof Mode.enum; +export const ModeEnum = Mode.enum; + export type PickVersionValues = z.infer; export const PickVersionValues = z.enum(['BASE', 'CURRENT', 'TARGET', 'MERGED']); export type PickVersionValuesEnum = typeof PickVersionValues.enum; export const PickVersionValuesEnum = PickVersionValues.enum; +// Specific handling of special fields according to: +// https://github.com/elastic/kibana/issues/186544 +export const FIELDS_TO_UPGRADE_TO_CURRENT_VERSION = [ + 'enabled', + 'exceptions_list', + 'alert_suppression', + 'actions', + 'throttle', + 'response_actions', + 'meta', + 'output_index', + 'namespace', + 'alias_purpose', + 'alias_target_id', + 'outcome', + 'concurrent_searches', + 'items_per_search', +] as const; + +export const NON_UPGRADEABLE_DIFFABLE_FIELDS = [ + 'type', + 'rule_id', + 'version', + 'author', + 'license', +] as const; + +type NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE = { + readonly [key in (typeof NON_UPGRADEABLE_DIFFABLE_FIELDS)[number]]: true; +}; + +// This transformation is needed to have Zod's `omit` accept the rule fields that need to be omitted +export const DiffableFieldsToOmit = NON_UPGRADEABLE_DIFFABLE_FIELDS.reduce((acc, field) => { + return { ...acc, [field]: true }; +}, {} as NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE); + /** * Fields upgradable by the /upgrade/_perform endpoint. * Specific fields are omitted because they are not upgradeable, and @@ -23,18 +64,12 @@ export const PickVersionValuesEnum = PickVersionValues.enum; * See: https://github.com/elastic/kibana/issues/186544 */ export type DiffableUpgradableFields = z.infer; -export const DiffableUpgradableFields = DiffableAllFields.omit({ - type: true, - rule_id: true, - version: true, - author: true, - license: true, -}); +export const DiffableUpgradableFields = DiffableAllFields.omit(DiffableFieldsToOmit); export type FieldUpgradeSpecifier = z.infer< ReturnType>> >; -const fieldUpgradeSpecifier = (fieldSchema: T) => +export const fieldUpgradeSpecifier = (fieldSchema: T) => z.discriminatedUnion('pick_version', [ z .object({ diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts index a5596ca4c8498..86809a3a79a93 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts @@ -13,7 +13,7 @@ import { buildSiemResponse } from '../../../routes/utils'; import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client'; import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client'; import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad'; -import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets'; +import { getRuleGroups } from '../../model/rule_groups/get_rule_groups'; export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter) => { router.versioned @@ -44,7 +44,7 @@ export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter ruleObjectsClient, }); const { currentRules, installableRules, upgradeableRules, totalAvailableRules } = - getVersionBuckets(ruleVersionsMap); + getRuleGroups(ruleVersionsMap); const body: GetPrebuiltRulesStatusResponseBody = { stats: { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts index 8ffec60a26c11..1a29568ca496b 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts @@ -25,9 +25,9 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt import { createPrebuiltRules } from '../../logic/rule_objects/create_prebuilt_rules'; import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client'; import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad'; -import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets'; import { performTimelinesInstallation } from '../../logic/perform_timelines_installation'; import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants'; +import { getRuleGroups } from '../../model/rule_groups/get_rule_groups'; export const performRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => { router.versioned @@ -80,7 +80,7 @@ export const performRuleInstallationRoute = (router: SecuritySolutionPluginRoute ruleObjectsClient, versionSpecifiers: mode === 'ALL_RULES' ? undefined : request.body.rules, }); - const { currentRules, installableRules } = getVersionBuckets(ruleVersionsMap); + const { currentRules, installableRules } = getRuleGroups(ruleVersionsMap); // Perform all the checks we can before we start the upgrade process if (mode === 'SPECIFIC_RULES') { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts new file mode 100644 index 0000000000000..a7ff15a82a3db --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts @@ -0,0 +1,60 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { assertDiffableFieldsMatchRuleType } from './assert_diffable_fields_match_rule_type'; +import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine'; + +describe('assertDiffableFieldsMatchRuleType', () => { + describe('valid scenarios -', () => { + it('should validate all fields in DIFFABLE_RULE_TYPE_FIELDS_MAP', () => { + DIFFABLE_RULE_TYPE_FIELDS_MAP.forEach((fields, ruleType) => { + expect(() => { + assertDiffableFieldsMatchRuleType(fields, ruleType); + }).not.toThrow(); + }); + }); + + it('should not throw an error for valid upgradeable fields', () => { + expect(() => { + assertDiffableFieldsMatchRuleType(['name', 'description', 'severity'], 'query'); + }).not.toThrow(); + }); + + it('should handle valid rule type correctly', () => { + expect(() => { + assertDiffableFieldsMatchRuleType(['eql_query'], 'eql'); + }).not.toThrow(); + }); + + it('should handle empty upgradeable fields array', () => { + expect(() => { + assertDiffableFieldsMatchRuleType([], 'query'); + }).not.toThrow(); + }); + }); + + describe('invalid scenarios -', () => { + it('should throw an error for invalid upgradeable fields', () => { + expect(() => { + assertDiffableFieldsMatchRuleType(['invalid_field'], 'query'); + }).toThrow("invalid_field is not a valid upgradeable field for type 'query'"); + }); + + it('should throw for incompatible rule types', () => { + expect(() => { + assertDiffableFieldsMatchRuleType(['eql_query'], 'query'); + }).toThrow("eql_query is not a valid upgradeable field for type 'query'"); + }); + + it('should throw an error for an unknown rule type', () => { + expect(() => { + // @ts-expect-error - unknown rule + assertDiffableFieldsMatchRuleType(['name'], 'unknown_type'); + }).toThrow(); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts new file mode 100644 index 0000000000000..14ac905ca885d --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { DiffableRuleTypes } from '../../../../../../common/api/detection_engine'; +import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine'; + +/** + * Validates that the upgradeable (diffable) fields match the target rule type's diffable fields. + * + * This function is used in the rule upgrade process to ensure that the fields + * specified for upgrade in the request body are valid for the target rule type. + * It checks each upgradeable field provided in body.rule[].fields against the + * set of diffable fields for the target rule type. + * + * @param {string[]} diffableFields - An array of field names to be upgraded. + * @param {string} ruleType - A rule type (e.g., 'query', 'eql', 'machine_learning'). + * @throws {Error} If an upgradeable field is not valid for the target rule type. + * + * @examples + * assertDiffableFieldsMatchRuleType(['kql_query', 'severity'], 'query'); + * assertDiffableFieldsMatchRuleType(['esql_query', 'description'], 'esql'); + * assertDiffableFieldsMatchRuleType(['machine_learning_job_id'], 'eql'); // throws error + * + * @see {@link DIFFABLE_RULE_TYPE_FIELDS_MAP} in diffable_rule.ts for the mapping of rule types to their diffable fields. + */ +export const assertDiffableFieldsMatchRuleType = ( + diffableFields: string[], + ruleType: DiffableRuleTypes +) => { + const diffableFieldsForType = new Set(DIFFABLE_RULE_TYPE_FIELDS_MAP.get(ruleType)); + for (const diffableField of diffableFields) { + if (!diffableFieldsForType.has(diffableField)) { + throw new Error(`${diffableField} is not a valid upgradeable field for type '${ruleType}'`); + } + } +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.test.ts new file mode 100644 index 0000000000000..d4cd1ae010067 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.test.ts @@ -0,0 +1,131 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { assertPickVersionIsTarget } from './assert_pick_version_is_target'; +import type { + PerformRuleUpgradeRequestBody, + PickVersionValues, +} from '../../../../../../common/api/detection_engine'; + +describe('assertPickVersionIsTarget', () => { + const ruleId = 'test-rule-id'; + const createExpectedError = (id: string) => + `Rule update for rule ${id} has a rule type change. All 'pick_version' values for rule must match 'TARGET'`; + + describe('valid cases - ', () => { + it('should not throw when pick_version is TARGET for ALL_RULES mode', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).not.toThrow(); + }); + + it('should not throw when pick_version is TARGET for SPECIFIC_RULES mode', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'SPECIFIC_RULES', + rules: [ + { + rule_id: ruleId, + revision: 1, + version: 1, + pick_version: 'TARGET', + }, + ], + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).not.toThrow(); + }); + + it('should not throw when all pick_version values are TARGET', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'SPECIFIC_RULES', + pick_version: 'TARGET', + rules: [ + { + rule_id: ruleId, + revision: 1, + version: 1, + pick_version: 'TARGET', + fields: { + name: { pick_version: 'TARGET' }, + description: { pick_version: 'TARGET' }, + }, + }, + ], + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).not.toThrow(); + }); + }); + + describe('invalid cases - ', () => { + it('should throw when pick_version is not TARGET for ALL_RULES mode', () => { + const pickVersions: PickVersionValues[] = ['BASE', 'CURRENT', 'MERGED']; + + pickVersions.forEach((pickVersion) => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'ALL_RULES', + pick_version: pickVersion, + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).toThrowError( + createExpectedError(ruleId) + ); + }); + }); + + it('should throw when pick_version is not TARGET for SPECIFIC_RULES mode', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'SPECIFIC_RULES', + rules: [ + { + rule_id: ruleId, + revision: 1, + version: 1, + pick_version: 'BASE', + }, + ], + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).toThrowError( + createExpectedError(ruleId) + ); + }); + + it('should throw when any field-specific pick_version is not TARGET', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'SPECIFIC_RULES', + rules: [ + { + rule_id: ruleId, + revision: 1, + version: 1, + pick_version: 'TARGET', + fields: { + name: { pick_version: 'BASE' }, + }, + }, + ], + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).toThrowError( + createExpectedError(ruleId) + ); + }); + + it('should throw when pick_version is missing (defaults to MERGED)', () => { + const requestBody: PerformRuleUpgradeRequestBody = { + mode: 'SPECIFIC_RULES', + rules: [{ rule_id: ruleId, revision: 1, version: 1 }], + }; + + expect(() => assertPickVersionIsTarget({ requestBody, ruleId })).toThrow(); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.ts new file mode 100644 index 0000000000000..63e67512be249 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_pick_version_is_target.ts @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import type { + PerformRuleUpgradeRequestBody, + PickVersionValues, +} from '../../../../../../common/api/detection_engine'; + +interface AssertRuleTypeMatchProps { + requestBody: PerformRuleUpgradeRequestBody; + ruleId: string; +} + +/* + * Assert that, in the case where the rule is undergoing a rule type change, + * the pick_version value is set to 'TARGET' at all levels (global, rule-specific and field-specific) + */ +export const assertPickVersionIsTarget = ({ requestBody, ruleId }: AssertRuleTypeMatchProps) => { + const pickVersions: Array = []; + + if (requestBody.mode === 'SPECIFIC_RULES') { + const rulePayload = requestBody.rules.find((rule) => rule.rule_id === ruleId); + + // Rule-level pick_version overrides global pick_version. Pick rule-level pick_version if it + // exists, otherwise use global pick_version. If none exist, we default to 'MERGED'. + pickVersions.push(rulePayload?.pick_version ?? requestBody.pick_version ?? 'MERGED'); + + if (rulePayload?.fields) { + const fieldPickValues = Object.values(rulePayload?.fields).map((field) => field.pick_version); + pickVersions.push(...fieldPickValues); + } + } else { + // mode: ALL_RULES + pickVersions.push(requestBody.pick_version ?? 'MERGED'); + } + + const allPickVersionsAreTarget = pickVersions.every((version) => version === 'TARGET'); + + // If pick_version is provided at any levels, they must all be set to 'TARGET' + if (!allPickVersionsAreTarget) { + throw new Error( + `Rule update for rule ${ruleId} has a rule type change. All 'pick_version' values for rule must match 'TARGET'` + ); + } +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.test.ts new file mode 100644 index 0000000000000..ac5db9ef1e7f2 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.test.ts @@ -0,0 +1,118 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { createFieldUpgradeSpecifier } from './create_field_upgrade_specifier'; +import { + PickVersionValuesEnum, + type DiffableRuleTypes, +} from '../../../../../../common/api/detection_engine'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; + +describe('createFieldUpgradeSpecifier', () => { + const defaultArgs = { + fieldName: 'name' as keyof PrebuiltRuleAsset, + globalPickVersion: PickVersionValuesEnum.MERGED, + ruleId: 'rule-1', + targetRuleType: 'query' as DiffableRuleTypes, + }; + + it('should return rule-specific pick version when no specific fields are defined', () => { + const result = createFieldUpgradeSpecifier({ + ...defaultArgs, + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + pick_version: PickVersionValuesEnum.BASE, + revision: 1, + version: 1, + }, + }); + expect(result).toEqual({ pick_version: PickVersionValuesEnum.BASE }); + }); + + it('should return field-specific pick version when defined', () => { + const result = createFieldUpgradeSpecifier({ + ...defaultArgs, + fieldName: 'description', + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + pick_version: PickVersionValuesEnum.TARGET, + revision: 1, + version: 1, + fields: { description: { pick_version: PickVersionValuesEnum.CURRENT } }, + }, + }); + expect(result).toEqual({ + pick_version: PickVersionValuesEnum.CURRENT, + }); + }); + + it('should return resolved value for specifc fields with RESOLVED pick versions', () => { + const result = createFieldUpgradeSpecifier({ + ...defaultArgs, + fieldName: 'description', + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + revision: 1, + version: 1, + fields: { + description: { pick_version: 'RESOLVED', resolved_value: 'New description' }, + }, + }, + }); + expect(result).toEqual({ + pick_version: 'RESOLVED', + resolved_value: 'New description', + }); + }); + + it('should handle fields that require mapping', () => { + const result = createFieldUpgradeSpecifier({ + ...defaultArgs, + fieldName: 'index' as keyof PrebuiltRuleAsset, + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + revision: 1, + version: 1, + fields: { data_source: { pick_version: PickVersionValuesEnum.CURRENT } }, + }, + }); + expect(result).toEqual({ pick_version: PickVersionValuesEnum.CURRENT }); + }); + + it('should fall back to rule-level pick version when field is not specified', () => { + const result = createFieldUpgradeSpecifier({ + ...defaultArgs, + fieldName: 'description', + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + pick_version: PickVersionValuesEnum.TARGET, + revision: 1, + version: 1, + fields: { name: { pick_version: PickVersionValuesEnum.CURRENT } }, + }, + }); + expect(result).toEqual({ + pick_version: PickVersionValuesEnum.TARGET, + }); + }); + + it('should throw error if field is not a valid upgradeable field', () => { + // machine_learning_job_id field does not match 'eql' target rule type + expect(() => + createFieldUpgradeSpecifier({ + ...defaultArgs, + targetRuleType: 'eql', + ruleUpgradeSpecifier: { + rule_id: 'rule-1', + revision: 1, + version: 1, + fields: { machine_learning_job_id: { pick_version: PickVersionValuesEnum.CURRENT } }, + }, + }) + ).toThrowError(`machine_learning_job_id is not a valid upgradeable field for type 'eql'`); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.ts new file mode 100644 index 0000000000000..7526394c5a75f --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_field_upgrade_specifier.ts @@ -0,0 +1,72 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import { assertDiffableFieldsMatchRuleType } from './assert_diffable_fields_match_rule_type'; +import { + type UpgradeSpecificRulesRequest, + type RuleFieldsToUpgrade, + type DiffableRuleTypes, + type FieldUpgradeSpecifier, + type PickVersionValues, +} from '../../../../../../common/api/detection_engine'; +import { type PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import { mapRuleFieldToDiffableRuleField } from './diffable_rule_fields_mappings'; + +interface CreateFieldUpgradeSpecifierArgs { + fieldName: keyof PrebuiltRuleAsset; + ruleUpgradeSpecifier: UpgradeSpecificRulesRequest['rules'][number]; + targetRuleType: DiffableRuleTypes; + globalPickVersion: PickVersionValues; +} + +/** + * Creates a field upgrade specifier for a given field in PrebuiltRuleAsset. + * + * This function determines how a specific field should be upgraded based on the + * upgrade request body and the pick_version at global, rule and field-levels, + * when the mode is SPECIFIC_RULES. + */ +export const createFieldUpgradeSpecifier = ({ + fieldName, + ruleUpgradeSpecifier, + targetRuleType, + globalPickVersion, +}: CreateFieldUpgradeSpecifierArgs): FieldUpgradeSpecifier => { + if (!ruleUpgradeSpecifier.fields || Object.keys(ruleUpgradeSpecifier.fields).length === 0) { + return { + pick_version: ruleUpgradeSpecifier.pick_version ?? globalPickVersion, + }; + } + + assertDiffableFieldsMatchRuleType(Object.keys(ruleUpgradeSpecifier.fields), targetRuleType); + + const fieldsToUpgradePayload = ruleUpgradeSpecifier.fields as Record< + string, + RuleFieldsToUpgrade[keyof RuleFieldsToUpgrade] + >; + + const fieldGroup = mapRuleFieldToDiffableRuleField({ + ruleType: targetRuleType, + fieldName, + }); + + const fieldUpgradeSpecifier = fieldsToUpgradePayload[fieldGroup]; + + if (fieldUpgradeSpecifier?.pick_version === 'RESOLVED') { + return { + pick_version: 'RESOLVED', + resolved_value: fieldUpgradeSpecifier.resolved_value, + }; + } + + return { + pick_version: + // If there's no matching specific field upgrade specifier in the payload, + // we fallback to a rule level pick_version. Since this is also optional, + // we default to the global pick_version. + fieldUpgradeSpecifier?.pick_version ?? ruleUpgradeSpecifier.pick_version ?? globalPickVersion, + }; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_props_to_rule_type_map.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_props_to_rule_type_map.ts new file mode 100644 index 0000000000000..d0b798fabaeb6 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_props_to_rule_type_map.ts @@ -0,0 +1,43 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { + SharedCreateProps, + TypeSpecificCreatePropsInternal, +} from '../../../../../../common/api/detection_engine'; +import { type PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; + +function createRuleTypeToCreateRulePropsMap() { + // SharedCreateProps is an extension of BaseCreateProps, but includes rule_id + const baseFields = Object.keys(SharedCreateProps.shape); + + return new Map( + TypeSpecificCreatePropsInternal.options.map((option) => { + const typeName = option.shape.type.value; + const typeSpecificFieldsForType = Object.keys(option.shape); + + return [typeName, [...baseFields, ...typeSpecificFieldsForType] as [keyof PrebuiltRuleAsset]]; + }) + ); +} + +/** + * Map of the CreateProps field names, by rule type. + * + * Helps creating the payload to be passed to the `upgradePrebuiltRules()` method during the + * Upgrade workflow (`/upgrade/_perform` endpoint) + * + * Creating this Map dynamically, based on BaseCreateProps and TypeSpecificFields, ensures that we don't need to: + * - manually add rule types to this Map if they are created + * - manually add or remove any fields if they are added or removed to a specific rule type + * - manually add or remove any fields if we decide that they should not be part of the upgradable fields. + * + * Notice that this Map includes, for each rule type, all fields that are part of the BaseCreateProps and all fields that + * are part of the TypeSpecificFields, including those that are not part of RuleUpgradeSpecifierFields schema, where + * the user of the /upgrade/_perform endpoint can specify which fields to upgrade during the upgrade workflow. + */ +export const FIELD_NAMES_BY_RULE_TYPE_MAP = createRuleTypeToCreateRulePropsMap(); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_upgradeable_rules_payload.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_upgradeable_rules_payload.ts new file mode 100644 index 0000000000000..97e587646e524 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/create_upgradeable_rules_payload.ts @@ -0,0 +1,145 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import { pickBy } from 'lodash'; +import type { PromisePoolError } from '../../../../../utils/promise_pool'; +import { + PickVersionValuesEnum, + type PerformRuleUpgradeRequestBody, + type PickVersionValues, + type AllFieldsDiff, + MissingVersion, +} from '../../../../../../common/api/detection_engine'; +import { convertRuleToDiffable } from '../../../../../../common/detection_engine/prebuilt_rules/diff/convert_rule_to_diffable'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import { assertPickVersionIsTarget } from './assert_pick_version_is_target'; +import { FIELD_NAMES_BY_RULE_TYPE_MAP } from './create_props_to_rule_type_map'; +import { calculateRuleFieldsDiff } from '../../logic/diff/calculation/calculate_rule_fields_diff'; +import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; +import { getValueForField } from './get_value_for_field'; + +interface CreateModifiedPrebuiltRuleAssetsProps { + upgradeableRules: RuleTriad[]; + requestBody: PerformRuleUpgradeRequestBody; +} + +interface ProcessedRules { + modifiedPrebuiltRuleAssets: PrebuiltRuleAsset[]; + processingErrors: Array>; +} + +export const createModifiedPrebuiltRuleAssets = ({ + upgradeableRules, + requestBody, +}: CreateModifiedPrebuiltRuleAssetsProps) => { + const { pick_version: globalPickVersion = PickVersionValuesEnum.MERGED, mode } = requestBody; + + const { modifiedPrebuiltRuleAssets, processingErrors } = upgradeableRules.reduce( + (processedRules, upgradeableRule) => { + const targetRuleType = upgradeableRule.target.type; + const ruleId = upgradeableRule.target.rule_id; + const fieldNames = FIELD_NAMES_BY_RULE_TYPE_MAP.get(targetRuleType); + + try { + if (fieldNames === undefined) { + throw new Error(`Unexpected rule type: ${targetRuleType}`); + } + + const { current, target } = upgradeableRule; + if (current.type !== target.type) { + assertPickVersionIsTarget({ ruleId, requestBody }); + } + + const calculatedRuleDiff = calculateRuleFieldsDiff({ + base_version: upgradeableRule.base + ? convertRuleToDiffable(convertPrebuiltRuleAssetToRuleResponse(upgradeableRule.base)) + : MissingVersion, + current_version: convertRuleToDiffable(upgradeableRule.current), + target_version: convertRuleToDiffable( + convertPrebuiltRuleAssetToRuleResponse(upgradeableRule.target) + ), + }) as AllFieldsDiff; + + if (mode === 'ALL_RULES' && globalPickVersion === 'MERGED') { + const fieldsWithConflicts = Object.keys(getFieldsDiffConflicts(calculatedRuleDiff)); + if (fieldsWithConflicts.length > 0) { + // If the mode is ALL_RULES, no fields can be overriden to any other pick_version + // than "MERGED", so throw an error for the fields that have conflicts. + throw new Error( + `Merge conflicts found in rule '${ruleId}' for fields: ${fieldsWithConflicts.join( + ', ' + )}. Please resolve the conflict manually or choose another value for 'pick_version'` + ); + } + } + + const modifiedPrebuiltRuleAsset = createModifiedPrebuiltRuleAsset({ + upgradeableRule, + fieldNames, + requestBody, + globalPickVersion, + calculatedRuleDiff, + }); + + processedRules.modifiedPrebuiltRuleAssets.push(modifiedPrebuiltRuleAsset); + + return processedRules; + } catch (err) { + processedRules.processingErrors.push({ + error: err, + item: { rule_id: ruleId }, + }); + + return processedRules; + } + }, + { + modifiedPrebuiltRuleAssets: [], + processingErrors: [], + } + ); + + return { + modifiedPrebuiltRuleAssets, + processingErrors, + }; +}; + +interface CreateModifiedPrebuiltRuleAssetParams { + upgradeableRule: RuleTriad; + fieldNames: Array; + globalPickVersion: PickVersionValues; + requestBody: PerformRuleUpgradeRequestBody; + calculatedRuleDiff: AllFieldsDiff; +} + +function createModifiedPrebuiltRuleAsset({ + upgradeableRule, + fieldNames, + globalPickVersion, + requestBody, + calculatedRuleDiff, +}: CreateModifiedPrebuiltRuleAssetParams): PrebuiltRuleAsset { + const modifiedPrebuiltRuleAsset = {} as Record; + + for (const fieldName of fieldNames) { + modifiedPrebuiltRuleAsset[fieldName] = getValueForField({ + fieldName, + upgradeableRule, + globalPickVersion, + requestBody, + ruleFieldsDiff: calculatedRuleDiff, + }); + } + + return modifiedPrebuiltRuleAsset as PrebuiltRuleAsset; +} + +const getFieldsDiffConflicts = (ruleFieldsDiff: Partial) => + pickBy(ruleFieldsDiff, (diff) => { + return diff.conflict !== 'NONE'; + }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/diffable_rule_fields_mappings.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/diffable_rule_fields_mappings.ts new file mode 100644 index 0000000000000..d56747f9db264 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/diffable_rule_fields_mappings.ts @@ -0,0 +1,211 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import { get } from 'lodash'; +import type { + RuleSchedule, + InlineKqlQuery, + ThreeWayDiff, + DiffableRuleTypes, +} from '../../../../../../common/api/detection_engine'; +import { type AllFieldsDiff } from '../../../../../../common/api/detection_engine'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; + +/** + * Retrieves and transforms the value for a specific field from a DiffableRule group. + * + * Maps PrebuiltRuleAsset schema fields to their corresponding DiffableRule group values. It also + * applies necessary transformations to ensure the returned value matches the expected format + * for the PrebuiltRuleAsset schema. + * + * @param {keyof PrebuiltRuleAsset} field - The field name in the PrebuiltRuleAsset schema. + * @param {ThreeWayDiff['merged_version']} diffableField - The corresponding field value from the DiffableRule. + * + * @example + * // For an 'index' field + * mapDiffableRuleFieldValueToRuleSchema('index', { index_patterns: ['logs-*'] }) + * // Returns: ['logs-*'] + * + * @example + * // For a 'from' field in a rule schedule + * mapDiffableRuleFieldValueToRuleSchema('from', { interval: '5d', lookback: '30d' }) + * // Returns: 'now-30d' + * + */ +export const mapDiffableRuleFieldValueToRuleSchemaFormat = ( + fieldName: keyof PrebuiltRuleAsset, + diffableField: ThreeWayDiff['merged_version'] +) => { + const diffableRuleSubfieldName = mapRuleFieldToDiffableRuleSubfield(fieldName); + + const transformedValue = transformDiffableFieldValues(fieldName, diffableField); + if (transformedValue.type === 'TRANSFORMED_FIELD') { + return transformedValue.value; + } + + // From the ThreeWayDiff, get the specific field that maps to the diffable rule field + // Otherwise, the diffableField itself already matches the rule field, so retrieve that value. + const mappedField = get(diffableField, diffableRuleSubfieldName, diffableField); + + return mappedField; +}; + +interface MapRuleFieldToDiffableRuleFieldParams { + ruleType: DiffableRuleTypes; + fieldName: string; +} +/** + * Maps a PrebuiltRuleAsset schema field name to its corresponding DiffableRule group. + * + * Determines which group in the DiffableRule schema a given field belongs to. Handles special + * cases for query-related fields based on the rule type. + * + * @param {string} fieldName - The field name from the PrebuiltRuleAsset schema. + * @param {string} ruleType - The type of the rule being processed. + * + * @example + * mapRuleFieldToDiffableRuleField('index', 'query') + * // Returns: 'data_source' + * + * @example + * mapRuleFieldToDiffableRuleField('query', 'eql') + * // Returns: 'eql_query' + * + */ +export function mapRuleFieldToDiffableRuleField({ + ruleType, + fieldName, +}: MapRuleFieldToDiffableRuleFieldParams): keyof AllFieldsDiff { + const diffableRuleFieldMap: Record = { + building_block_type: 'building_block', + saved_id: 'kql_query', + threat_query: 'threat_query', + threat_language: 'threat_query', + threat_filters: 'threat_query', + index: 'data_source', + data_view_id: 'data_source', + rule_name_override: 'rule_name_override', + interval: 'rule_schedule', + from: 'rule_schedule', + to: 'rule_schedule', + timeline_id: 'timeline_template', + timeline_title: 'timeline_template', + timestamp_override: 'timestamp_override', + timestamp_override_fallback_disabled: 'timestamp_override', + }; + + // Handle query, filters and language fields based on rule type + if (fieldName === 'query' || fieldName === 'language' || fieldName === 'filters') { + switch (ruleType) { + case 'query': + case 'saved_query': + return 'kql_query' as const; + case 'eql': + return 'eql_query'; + case 'esql': + return 'esql_query'; + default: + return 'kql_query'; + } + } + + return diffableRuleFieldMap[fieldName] || fieldName; +} + +/** + * Maps a PrebuiltRuleAsset schema field name to its corresponding property + * name within a DiffableRule group. + * + * @param {string} fieldName - The field name from the PrebuiltRuleAsset schema. + * @returns {string} The corresponding property name in the DiffableRule group. + * + * @example + * mapRuleFieldToDiffableRuleSubfield('index') + * // Returns: 'index_patterns' + * + * @example + * mapRuleFieldToDiffableRuleSubfield('from') + * // Returns: 'lookback' + * + */ +export function mapRuleFieldToDiffableRuleSubfield(fieldName: string): string { + const fieldMapping: Record = { + index: 'index_patterns', + data_view_id: 'data_view_id', + saved_id: 'saved_query_id', + building_block_type: 'type', + rule_name_override: 'field_name', + timestamp_override: 'field_name', + timestamp_override_fallback_disabled: 'fallback_disabled', + timeline_id: 'timeline_id', + timeline_title: 'timeline_title', + interval: 'interval', + from: 'lookback', + to: 'lookback', + }; + + return fieldMapping[fieldName] || fieldName; +} + +type TransformValuesReturnType = + | { + type: 'TRANSFORMED_FIELD'; + value: unknown; + } + | { type: 'NON_TRANSFORMED_FIELD' }; + +/** + * Transforms specific field values from the DiffableRule format to the PrebuiltRuleAsset/RuleResponse format. + * + * This function is used in the rule upgrade process to ensure that certain fields + * are correctly formatted when creating the updated rules payload. It handles + * special cases where the format differs between the DiffableRule and the + * PrebuiltRuleAsset/RuleResponse schemas. + * + * @param {string} fieldName - The name of the field being processed. + * @param {RuleSchedule | InlineKqlQuery | unknown} diffableFieldValue - The value of the field in DiffableRule format. + * + * @returns {TransformValuesReturnType} An object indicating whether the field was transformed + * and its new value if applicable. + * - If transformed: { type: 'TRANSFORMED_FIELD', value: transformedValue } + * - If not transformed: { type: 'NON_TRANSFORMED_FIELD' } + * + * @example + * // Transforms 'from' field + * transformDiffableFieldValues('from', { lookback: '30d' }) + * // Returns: { type: 'TRANSFORMED_FIELD', value: 'now-30d' } + * + * @example + * // Transforms 'saved_id' field for inline queries + * transformDiffableFieldValues('saved_id', { type: 'inline_query', ... }) + * // Returns: { type: 'TRANSFORMED_FIELD', value: undefined } + * + */ +export const transformDiffableFieldValues = ( + fieldName: string, + diffableFieldValue: RuleSchedule | InlineKqlQuery | unknown +): TransformValuesReturnType => { + if (fieldName === 'from' && isRuleSchedule(diffableFieldValue)) { + return { type: 'TRANSFORMED_FIELD', value: `now-${diffableFieldValue.lookback}` }; + } else if (fieldName === 'to') { + return { type: 'TRANSFORMED_FIELD', value: `now` }; + } else if (fieldName === 'saved_id' && isInlineQuery(diffableFieldValue)) { + // saved_id should be set only for rules with SavedKqlQuery, undefined otherwise + return { type: 'TRANSFORMED_FIELD', value: undefined }; + } + + return { type: 'NON_TRANSFORMED_FIELD' }; +}; + +function isRuleSchedule(value: unknown): value is RuleSchedule { + return typeof value === 'object' && value !== null && 'lookback' in value; +} + +function isInlineQuery(value: unknown): value is InlineKqlQuery { + return ( + typeof value === 'object' && value !== null && 'type' in value && value.type === 'inline_query' + ); +} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.test.ts new file mode 100644 index 0000000000000..9a1ca051c54fa --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.test.ts @@ -0,0 +1,65 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { getFieldPredefinedValue } from './get_field_predefined_value'; +import { + NON_UPGRADEABLE_DIFFABLE_FIELDS, + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION, +} from '../../../../../../common/api/detection_engine'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; + +describe('getFieldPredefinedValue', () => { + const mockUpgradeableRule = { + current: { + rule_id: 'current_rule_id', + type: 'query', + enabled: true, + name: 'Current Rule', + description: 'Current description', + version: 1, + author: ['Current Author'], + license: 'Current License', + }, + target: { + rule_id: 'target_rule_id', + type: 'query', + enabled: false, + name: 'Target Rule', + description: 'Target description', + version: 2, + author: ['Target Author'], + license: 'Target License', + }, + } as RuleTriad; + + it('should return PREDEFINED_VALUE with target value for fields in NON_UPGRADEABLE_DIFFABLE_FIELDS', () => { + NON_UPGRADEABLE_DIFFABLE_FIELDS.forEach((field) => { + const result = getFieldPredefinedValue(field as keyof PrebuiltRuleAsset, mockUpgradeableRule); + expect(result).toEqual({ + type: 'PREDEFINED_VALUE', + value: mockUpgradeableRule.target[field as keyof PrebuiltRuleAsset], + }); + }); + }); + + it('should return PREDEFINED_VALUE with current value for fields in FIELDS_TO_UPGRADE_TO_CURRENT_VERSION', () => { + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + const result = getFieldPredefinedValue(field as keyof PrebuiltRuleAsset, mockUpgradeableRule); + expect(result).toEqual({ + type: 'PREDEFINED_VALUE', + value: mockUpgradeableRule.current[field as keyof PrebuiltRuleAsset], + }); + }); + }); + + it('should return CUSTOMIZABLE_VALUE for fields not in NON_UPGRADEABLE_DIFFABLE_FIELDS or FIELDS_TO_UPGRADE_TO_CURRENT_VERSION', () => { + const upgradeableField = 'description'; + const result = getFieldPredefinedValue(upgradeableField, mockUpgradeableRule); + expect(result).toEqual({ type: 'CUSTOMIZABLE_VALUE' }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.ts new file mode 100644 index 0000000000000..777711e56470c --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_field_predefined_value.ts @@ -0,0 +1,73 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION, + NON_UPGRADEABLE_DIFFABLE_FIELDS, +} from '../../../../../../common/api/detection_engine'; +import { type PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; + +type GetFieldPredefinedValueReturnType = + | { + type: 'PREDEFINED_VALUE'; + value: unknown; + } + | { type: 'CUSTOMIZABLE_VALUE' }; + +/** + * Determines whether a field can be upgraded via API (i.e. whether it should take + * a predefined value or is customizable), and returns the value if it is predefined. + * + * This function checks whether a field can be upgraded via API contract and how it should + * be handled during the rule upgrade process. It uses the `NON_UPGRADEABLE_DIFFABLE_FIELDS` and + * `FIELDS_TO_UPGRADE_TO_CURRENT_VERSION` constants to make this determination. + * + * `NON_UPGRADEABLE_DIFFABLE_FIELDS` includes fields that are not upgradeable: 'type', 'rule_id', + * 'version', 'author', and 'license', and are always upgraded to the target version. + * + * `FIELDS_TO_UPGRADE_TO_CURRENT_VERSION` includes fields that should be updated to their + * current version, such as 'enabled', 'alert_suppression', 'actions', 'throttle', + * 'response_actions', 'meta', 'output_index', 'namespace', 'alias_purpose', + * 'alias_target_id', 'outcome', 'concurrent_searches', and 'items_per_search'. + * + * @param {keyof PrebuiltRuleAsset} fieldName - The field name to check for upgrade status. + * @param {RuleTriad} upgradeableRule - The rule object containing current and target versions. + * + * @returns {GetFieldPredefinedValueReturnType} An object indicating whether the field + * is upgradeable and its value to upgrade to if it's not upgradeable via API. + */ +export const getFieldPredefinedValue = ( + fieldName: keyof PrebuiltRuleAsset, + upgradeableRule: RuleTriad +): GetFieldPredefinedValueReturnType => { + if ( + NON_UPGRADEABLE_DIFFABLE_FIELDS.includes( + fieldName as (typeof NON_UPGRADEABLE_DIFFABLE_FIELDS)[number] + ) + ) { + return { + type: 'PREDEFINED_VALUE', + value: upgradeableRule.target[fieldName], + }; + } + + if ( + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.includes( + fieldName as (typeof FIELDS_TO_UPGRADE_TO_CURRENT_VERSION)[number] + ) + ) { + return { + type: 'PREDEFINED_VALUE', + value: upgradeableRule.current[fieldName], + }; + } + + return { + type: 'CUSTOMIZABLE_VALUE', + }; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.test.ts new file mode 100644 index 0000000000000..5b1c74825102c --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.test.ts @@ -0,0 +1,191 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { getUpgradeableRules } from './get_upgradeable_rules'; +import { ModeEnum, SkipRuleUpgradeReasonEnum } from '../../../../../../common/api/detection_engine'; +import type { + RuleResponse, + RuleUpgradeSpecifier, +} from '../../../../../../common/api/detection_engine'; +import { getPrebuiltRuleMockOfType } from '../../model/rule_assets/prebuilt_rule_asset.mock'; +import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; + +describe('getUpgradeableRules', () => { + const baseRule = getPrebuiltRuleMockOfType('query'); + const createUpgradeableRule = ( + ruleId: string, + currentVersion: number, + targetVersion: number + ): RuleTriad => { + return { + current: { + ...baseRule, + rule_id: ruleId, + version: currentVersion, + revision: 0, + }, + target: { ...baseRule, rule_id: ruleId, version: targetVersion }, + } as RuleTriad; + }; + + const mockUpgradeableRule = createUpgradeableRule('rule-1', 1, 2); + + const mockCurrentRule: RuleResponse = { + ...convertPrebuiltRuleAssetToRuleResponse(baseRule), + rule_id: 'rule-1', + revision: 0, + version: 1, + }; + + describe('ALL_RULES mode', () => { + it('should return all upgradeable rules when in ALL_RULES mode', () => { + const result = getUpgradeableRules({ + rawUpgradeableRules: [mockUpgradeableRule], + currentRules: [mockCurrentRule], + mode: ModeEnum.ALL_RULES, + }); + + expect(result.upgradeableRules).toEqual([mockUpgradeableRule]); + expect(result.fetchErrors).toEqual([]); + expect(result.skippedRules).toEqual([]); + }); + + it('should handle empty upgradeable rules list', () => { + const result = getUpgradeableRules({ + rawUpgradeableRules: [], + currentRules: [], + mode: ModeEnum.ALL_RULES, + }); + + expect(result.upgradeableRules).toEqual([]); + expect(result.fetchErrors).toEqual([]); + expect(result.skippedRules).toEqual([]); + }); + }); + + describe('SPECIFIC_RULES mode', () => { + const mockVersionSpecifier: RuleUpgradeSpecifier = { + rule_id: 'rule-1', + revision: 0, + version: 1, + }; + + it('should return specified upgradeable rules when in SPECIFIC_RULES mode', () => { + const result = getUpgradeableRules({ + rawUpgradeableRules: [mockUpgradeableRule], + currentRules: [mockCurrentRule], + versionSpecifiers: [mockVersionSpecifier], + mode: ModeEnum.SPECIFIC_RULES, + }); + + expect(result.upgradeableRules).toEqual([mockUpgradeableRule]); + expect(result.fetchErrors).toEqual([]); + expect(result.skippedRules).toEqual([]); + }); + + it('should handle rule not found', () => { + const result = getUpgradeableRules({ + rawUpgradeableRules: [mockUpgradeableRule], + currentRules: [mockCurrentRule], + versionSpecifiers: [{ ...mockVersionSpecifier, rule_id: 'nonexistent' }], + mode: ModeEnum.SPECIFIC_RULES, + }); + + expect(result.upgradeableRules).toEqual([mockUpgradeableRule]); + expect(result.fetchErrors).toHaveLength(1); + expect(result.fetchErrors[0].error.message).toContain( + 'Rule with rule_id "nonexistent" and version "1" not found' + ); + expect(result.skippedRules).toEqual([]); + }); + + it('should handle non-upgradeable rule', () => { + const nonUpgradeableRule: RuleResponse = { + ...convertPrebuiltRuleAssetToRuleResponse(baseRule), + rule_id: 'rule-2', + revision: 0, + version: 1, + }; + + const result = getUpgradeableRules({ + rawUpgradeableRules: [mockUpgradeableRule], + currentRules: [mockCurrentRule, nonUpgradeableRule], + versionSpecifiers: [mockVersionSpecifier, { ...mockVersionSpecifier, rule_id: 'rule-2' }], + mode: ModeEnum.SPECIFIC_RULES, + }); + + expect(result.upgradeableRules).toEqual([mockUpgradeableRule]); + expect(result.fetchErrors).toEqual([]); + expect(result.skippedRules).toEqual([ + { rule_id: 'rule-2', reason: SkipRuleUpgradeReasonEnum.RULE_UP_TO_DATE }, + ]); + }); + + it('should handle revision mismatch', () => { + const result = getUpgradeableRules({ + rawUpgradeableRules: [mockUpgradeableRule], + currentRules: [mockCurrentRule], + versionSpecifiers: [{ ...mockVersionSpecifier, revision: 1 }], + mode: ModeEnum.SPECIFIC_RULES, + }); + + expect(result.upgradeableRules).toEqual([]); + expect(result.fetchErrors).toHaveLength(1); + expect(result.fetchErrors[0].error.message).toContain( + 'Revision mismatch for rule_id rule-1: expected 0, got 1' + ); + expect(result.skippedRules).toEqual([]); + }); + + it('should handle multiple rules with mixed scenarios', () => { + const mockUpgradeableRule2 = createUpgradeableRule('rule-2', 1, 2); + const mockCurrentRule2: RuleResponse = { + ...convertPrebuiltRuleAssetToRuleResponse(baseRule), + rule_id: 'rule-2', + revision: 0, + version: 1, + }; + const mockCurrentRule3: RuleResponse = { + ...convertPrebuiltRuleAssetToRuleResponse(baseRule), + rule_id: 'rule-3', + revision: 1, + version: 1, + }; + + const result = getUpgradeableRules({ + rawUpgradeableRules: [ + mockUpgradeableRule, + mockUpgradeableRule2, + createUpgradeableRule('rule-3', 1, 2), + ], + currentRules: [mockCurrentRule, mockCurrentRule2, mockCurrentRule3], + versionSpecifiers: [ + mockVersionSpecifier, + { ...mockVersionSpecifier, rule_id: 'rule-2' }, + { ...mockVersionSpecifier, rule_id: 'rule-3', revision: 0 }, + { ...mockVersionSpecifier, rule_id: 'rule-4' }, + { ...mockVersionSpecifier, rule_id: 'rule-5', revision: 1 }, + ], + mode: ModeEnum.SPECIFIC_RULES, + }); + + expect(result.upgradeableRules).toEqual([mockUpgradeableRule, mockUpgradeableRule2]); + expect(result.fetchErrors).toHaveLength(3); + expect(result.fetchErrors[0].error.message).toContain( + 'Revision mismatch for rule_id rule-3: expected 1, got 0' + ); + expect(result.fetchErrors[1].error.message).toContain( + 'Rule with rule_id "rule-4" and version "1" not found' + ); + expect(result.fetchErrors[2].error.message).toContain( + 'Rule with rule_id "rule-5" and version "1" not found' + ); + expect(result.skippedRules).toEqual([]); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.ts new file mode 100644 index 0000000000000..acfdb674c309a --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_upgradeable_rules.ts @@ -0,0 +1,83 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { + RuleResponse, + RuleUpgradeSpecifier, + SkippedRuleUpgrade, +} from '../../../../../../common/api/detection_engine'; +import { ModeEnum, SkipRuleUpgradeReasonEnum } from '../../../../../../common/api/detection_engine'; +import type { PromisePoolError } from '../../../../../utils/promise_pool'; +import type { Mode } from '../../../../../../common/api/detection_engine/prebuilt_rules'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; + +export const getUpgradeableRules = ({ + rawUpgradeableRules, + currentRules, + versionSpecifiers, + mode, +}: { + rawUpgradeableRules: RuleTriad[]; + currentRules: RuleResponse[]; + versionSpecifiers?: RuleUpgradeSpecifier[]; + mode: Mode; +}) => { + const upgradeableRules = new Map( + rawUpgradeableRules.map((_rule) => [_rule.current.rule_id, _rule]) + ); + const fetchErrors: Array> = []; + const skippedRules: SkippedRuleUpgrade[] = []; + + if (mode === ModeEnum.SPECIFIC_RULES) { + const installedRuleIds = new Set(currentRules.map((rule) => rule.rule_id)); + const upgradeableRuleIds = new Set(rawUpgradeableRules.map(({ current }) => current.rule_id)); + versionSpecifiers?.forEach((rule) => { + // Check that the requested rule was found + if (!installedRuleIds.has(rule.rule_id)) { + fetchErrors.push({ + error: new Error( + `Rule with rule_id "${rule.rule_id}" and version "${rule.version}" not found` + ), + item: rule, + }); + return; + } + + // Check that the requested rule is upgradeable + if (!upgradeableRuleIds.has(rule.rule_id)) { + skippedRules.push({ + rule_id: rule.rule_id, + reason: SkipRuleUpgradeReasonEnum.RULE_UP_TO_DATE, + }); + return; + } + + // Check that rule revisions match (no update slipped in since the user reviewed the list) + const currentRevision = currentRules.find( + (currentRule) => currentRule.rule_id === rule.rule_id + )?.revision; + if (rule.revision !== currentRevision) { + fetchErrors.push({ + error: new Error( + `Revision mismatch for rule_id ${rule.rule_id}: expected ${currentRevision}, got ${rule.revision}` + ), + item: rule, + }); + // Remove the rule from the list of upgradeable rules + if (upgradeableRules.has(rule.rule_id)) { + upgradeableRules.delete(rule.rule_id); + } + } + }); + } + + return { + upgradeableRules: Array.from(upgradeableRules.values()), + fetchErrors, + skippedRules, + }; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_for_field.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_for_field.ts new file mode 100644 index 0000000000000..00de04c291aeb --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_for_field.ts @@ -0,0 +1,94 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { + PickVersionValues, + PerformRuleUpgradeRequestBody, + AllFieldsDiff, +} from '../../../../../../common/api/detection_engine'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; +import { createFieldUpgradeSpecifier } from './create_field_upgrade_specifier'; +import { mapDiffableRuleFieldValueToRuleSchemaFormat } from './diffable_rule_fields_mappings'; +import { getFieldPredefinedValue } from './get_field_predefined_value'; +import { getValueFromRuleTriad, getValueFromMergedVersion } from './get_value_from_rule_version'; + +interface GetValueForFieldArgs { + fieldName: keyof PrebuiltRuleAsset; + upgradeableRule: RuleTriad; + globalPickVersion: PickVersionValues; + requestBody: PerformRuleUpgradeRequestBody; + ruleFieldsDiff: AllFieldsDiff; +} + +export const getValueForField = ({ + fieldName, + upgradeableRule, + globalPickVersion, + requestBody, + ruleFieldsDiff, +}: GetValueForFieldArgs) => { + const fieldStatus = getFieldPredefinedValue(fieldName, upgradeableRule); + + if (fieldStatus.type === 'PREDEFINED_VALUE') { + return fieldStatus.value; + } + + if (requestBody.mode === 'ALL_RULES') { + return globalPickVersion === 'MERGED' + ? getValueFromMergedVersion({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier: { + pick_version: globalPickVersion, + }, + ruleFieldsDiff, + }) + : getValueFromRuleTriad({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier: { + pick_version: globalPickVersion, + }, + }); + } + + // Handle SPECIFIC_RULES mode + const ruleUpgradeSpecifier = requestBody.rules.find( + (r) => r.rule_id === upgradeableRule.target.rule_id + ); + + if (!ruleUpgradeSpecifier) { + throw new Error(`Rule payload for upgradable rule ${upgradeableRule.target.rule_id} not found`); + } + + const fieldUpgradeSpecifier = createFieldUpgradeSpecifier({ + fieldName, + ruleUpgradeSpecifier, + targetRuleType: upgradeableRule.target.type, + globalPickVersion, + }); + + if (fieldUpgradeSpecifier.pick_version === 'RESOLVED') { + const resolvedValue = fieldUpgradeSpecifier.resolved_value; + + return mapDiffableRuleFieldValueToRuleSchemaFormat(fieldName, resolvedValue); + } + + return fieldUpgradeSpecifier.pick_version === 'MERGED' + ? getValueFromMergedVersion({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier, + ruleFieldsDiff, + }) + : getValueFromRuleTriad({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier, + }); +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_from_rule_version.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_from_rule_version.ts new file mode 100644 index 0000000000000..3bef2ea7c742c --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/get_value_from_rule_version.ts @@ -0,0 +1,94 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { + RuleFieldsToUpgrade, + AllFieldsDiff, +} from '../../../../../../common/api/detection_engine'; +import { RULE_DEFAULTS } from '../../../rule_management/logic/detection_rules_client/mergers/apply_rule_defaults'; +import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; +import type { RuleTriad } from '../../model/rule_groups/get_rule_groups'; +import { + mapRuleFieldToDiffableRuleField, + mapDiffableRuleFieldValueToRuleSchemaFormat, +} from './diffable_rule_fields_mappings'; + +const RULE_DEFAULTS_FIELDS_SET = new Set(Object.keys(RULE_DEFAULTS)); + +export const getValueFromMergedVersion = ({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier, + ruleFieldsDiff, +}: { + fieldName: keyof PrebuiltRuleAsset; + upgradeableRule: RuleTriad; + fieldUpgradeSpecifier: NonNullable; + ruleFieldsDiff: AllFieldsDiff; +}) => { + const ruleId = upgradeableRule.target.rule_id; + const diffableRuleFieldName = mapRuleFieldToDiffableRuleField({ + ruleType: upgradeableRule.target.type, + fieldName, + }); + + if (fieldUpgradeSpecifier.pick_version === 'MERGED') { + const ruleFieldDiff = ruleFieldsDiff[diffableRuleFieldName]; + + if (ruleFieldDiff && ruleFieldDiff.conflict !== 'NONE') { + throw new Error( + `Automatic merge calculation for field '${diffableRuleFieldName}' in rule of rule_id ${ruleId} resulted in a conflict. Please resolve the conflict manually or choose another value for 'pick_version'.` + ); + } + + const mergedVersion = ruleFieldDiff.merged_version; + + return mapDiffableRuleFieldValueToRuleSchemaFormat(fieldName, mergedVersion); + } +}; + +export const getValueFromRuleTriad = ({ + fieldName, + upgradeableRule, + fieldUpgradeSpecifier, +}: { + fieldName: keyof PrebuiltRuleAsset; + upgradeableRule: RuleTriad; + fieldUpgradeSpecifier: NonNullable; +}) => { + const ruleId = upgradeableRule.target.rule_id; + const diffableRuleFieldName = mapRuleFieldToDiffableRuleField({ + ruleType: upgradeableRule.target.type, + fieldName, + }); + + const pickVersion = fieldUpgradeSpecifier.pick_version.toLowerCase() as keyof RuleTriad; + + // By this point, can be only 'base', 'current' or 'target' + const ruleVersion = upgradeableRule[pickVersion]; + + if (!ruleVersion) { + // Current and target versions should always be present + // but base version might not; throw if version is missing. + throw new Error( + `Missing '${pickVersion}' version for field '${diffableRuleFieldName}' in rule ${ruleId}` + ); + } + + // No need for conversions in the field names here since the rule versions in + // UpgradableRule have the values in the 'non-grouped' PrebuiltRuleAsset schema format. + const nonResolvedValue = ruleVersion[fieldName]; + + // If there's no value for the field in the rule versions, check if the field + // requires a default value for it. If it does, return the default value. + if (nonResolvedValue === undefined && RULE_DEFAULTS_FIELDS_SET.has(fieldName)) { + return RULE_DEFAULTS[fieldName as keyof typeof RULE_DEFAULTS]; + } + + // Otherwise, return the non-resolved value, which might be undefined. + return nonResolvedValue; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts index f95189d6af34d..085c41db3a5db 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts @@ -10,16 +10,10 @@ import { buildRouteValidationWithZod } from '@kbn/zod-helpers'; import { PERFORM_RULE_UPGRADE_URL, PerformRuleUpgradeRequestBody, - PickVersionValuesEnum, - SkipRuleUpgradeReasonEnum, + ModeEnum, } from '../../../../../../common/api/detection_engine/prebuilt_rules'; -import type { - PerformRuleUpgradeResponseBody, - SkippedRuleUpgrade, -} from '../../../../../../common/api/detection_engine/prebuilt_rules'; -import { assertUnreachable } from '../../../../../../common/utility_types'; +import type { PerformRuleUpgradeResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules'; import type { SecuritySolutionPluginRouter } from '../../../../../types'; -import type { PromisePoolError } from '../../../../../utils/promise_pool'; import { buildSiemResponse } from '../../../routes/utils'; import { aggregatePrebuiltRuleErrors } from '../../logic/aggregate_prebuilt_rule_errors'; import { performTimelinesInstallation } from '../../logic/perform_timelines_installation'; @@ -27,9 +21,10 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client'; import { upgradePrebuiltRules } from '../../logic/rule_objects/upgrade_prebuilt_rules'; import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad'; -import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; -import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets'; import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants'; +import { getUpgradeableRules } from './get_upgradeable_rules'; +import { createModifiedPrebuiltRuleAssets } from './create_upgradeable_rules_payload'; +import { getRuleGroups } from '../../model/rule_groups/get_rule_groups'; export const performRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) => { router.versioned @@ -63,108 +58,35 @@ export const performRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) => const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient); const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient); - const { mode, pick_version: globalPickVersion = PickVersionValuesEnum.TARGET } = - request.body; - - const fetchErrors: Array> = []; - const targetRules: PrebuiltRuleAsset[] = []; - const skippedRules: SkippedRuleUpgrade[] = []; + const { mode } = request.body; - const versionSpecifiers = mode === 'ALL_RULES' ? undefined : request.body.rules; - const versionSpecifiersMap = new Map( - versionSpecifiers?.map((rule) => [rule.rule_id, rule]) - ); - const ruleVersionsMap = await fetchRuleVersionsTriad({ + const versionSpecifiers = mode === ModeEnum.ALL_RULES ? undefined : request.body.rules; + const ruleTriadsMap = await fetchRuleVersionsTriad({ ruleAssetsClient, ruleObjectsClient, versionSpecifiers, }); - const versionBuckets = getVersionBuckets(ruleVersionsMap); - const { currentRules } = versionBuckets; - // The upgradeable rules list is mutable; we can remove rules from it because of version mismatch - let upgradeableRules = versionBuckets.upgradeableRules; + const ruleGroups = getRuleGroups(ruleTriadsMap); - // Perform all the checks we can before we start the upgrade process - if (mode === 'SPECIFIC_RULES') { - const installedRuleIds = new Set(currentRules.map((rule) => rule.rule_id)); - const upgradeableRuleIds = new Set( - upgradeableRules.map(({ current }) => current.rule_id) - ); - request.body.rules.forEach((rule) => { - // Check that the requested rule was found - if (!installedRuleIds.has(rule.rule_id)) { - fetchErrors.push({ - error: new Error( - `Rule with ID "${rule.rule_id}" and version "${rule.version}" not found` - ), - item: rule, - }); - return; - } - - // Check that the requested rule is upgradeable - if (!upgradeableRuleIds.has(rule.rule_id)) { - skippedRules.push({ - rule_id: rule.rule_id, - reason: SkipRuleUpgradeReasonEnum.RULE_UP_TO_DATE, - }); - return; - } - - // Check that rule revisions match (no update slipped in since the user reviewed the list) - const currentRevision = ruleVersionsMap.get(rule.rule_id)?.current?.revision; - if (rule.revision !== currentRevision) { - fetchErrors.push({ - error: new Error( - `Revision mismatch for rule ID ${rule.rule_id}: expected ${rule.revision}, got ${currentRevision}` - ), - item: rule, - }); - // Remove the rule from the list of upgradeable rules - upgradeableRules = upgradeableRules.filter( - ({ current }) => current.rule_id !== rule.rule_id - ); - } - }); - } + const { upgradeableRules, skippedRules, fetchErrors } = getUpgradeableRules({ + rawUpgradeableRules: ruleGroups.upgradeableRules, + currentRules: ruleGroups.currentRules, + versionSpecifiers, + mode, + }); - // Construct the list of target rule versions - upgradeableRules.forEach(({ current, target }) => { - const rulePickVersion = - versionSpecifiersMap?.get(current.rule_id)?.pick_version ?? globalPickVersion; - switch (rulePickVersion) { - case PickVersionValuesEnum.BASE: - const baseVersion = ruleVersionsMap.get(current.rule_id)?.base; - if (baseVersion) { - targetRules.push({ ...baseVersion, version: target.version }); - } else { - fetchErrors.push({ - error: new Error(`Could not find base version for rule ${current.rule_id}`), - item: current, - }); - } - break; - case PickVersionValuesEnum.CURRENT: - targetRules.push({ ...current, version: target.version }); - break; - case PickVersionValuesEnum.TARGET: - targetRules.push(target); - break; - case PickVersionValuesEnum.MERGED: - // TODO: Implement functionality to handle MERGED - targetRules.push(target); - break; - default: - assertUnreachable(rulePickVersion); + const { modifiedPrebuiltRuleAssets, processingErrors } = createModifiedPrebuiltRuleAssets( + { + upgradeableRules, + requestBody: request.body, } - }); + ); - // Perform the upgrade const { results: updatedRules, errors: installationErrors } = await upgradePrebuiltRules( detectionRulesClient, - targetRules + modifiedPrebuiltRuleAssets ); - const ruleErrors = [...fetchErrors, ...installationErrors]; + const ruleErrors = [...fetchErrors, ...processingErrors, ...installationErrors]; const { error: timelineInstallationError } = await performTimelinesInstallation( ctx.securitySolution diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts index ec3ca342bf8c9..00fc5e2beb5b8 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts @@ -17,9 +17,9 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client'; import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad'; import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; -import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets'; import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response'; import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants'; +import { getRuleGroups } from '../../model/rule_groups/get_rule_groups'; export const reviewRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => { router.versioned @@ -52,7 +52,7 @@ export const reviewRuleInstallationRoute = (router: SecuritySolutionPluginRouter ruleAssetsClient, ruleObjectsClient, }); - const { installableRules } = getVersionBuckets(ruleVersionsMap); + const { installableRules } = getRuleGroups(ruleVersionsMap); const body: ReviewRuleInstallationResponseBody = { stats: calculateRuleStats(installableRules), diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts index 8b229c6406b10..382ec27a1bf35 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts @@ -26,9 +26,9 @@ import { calculateRuleDiff } from '../../logic/diff/calculate_rule_diff'; import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client'; import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client'; import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad'; -import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets'; import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response'; import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants'; +import { getRuleGroups } from '../../model/rule_groups/get_rule_groups'; export const reviewRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) => { router.versioned @@ -61,7 +61,7 @@ export const reviewRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) => ruleAssetsClient, ruleObjectsClient, }); - const { upgradeableRules } = getVersionBuckets(ruleVersionsMap); + const { upgradeableRules } = getRuleGroups(ruleVersionsMap); const ruleDiffCalculationResults = upgradeableRules.map(({ current }) => { const ruleVersions = ruleVersionsMap.get(current.rule_id); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts index 3daaab8ecf10f..0dbfd8a230a5a 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts @@ -13,7 +13,7 @@ import { withSecuritySpan } from '../../../../../utils/with_security_span'; import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset'; import { validatePrebuiltRuleAssets } from './prebuilt_rule_assets_validation'; import { PREBUILT_RULE_ASSETS_SO_TYPE } from './prebuilt_rule_assets_type'; -import type { RuleVersionSpecifier } from '../../model/rule_versions/rule_version_specifier'; +import type { RuleVersionSpecifier } from '../rule_versions/rule_version_specifier'; const MAX_PREBUILT_RULES_COUNT = 10_000; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad.ts index ae7bdc6b391b4..11a5660e77a31 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad.ts @@ -8,7 +8,7 @@ import type { RuleVersions } from '../diff/calculate_rule_diff'; import type { IPrebuiltRuleAssetsClient } from '../rule_assets/prebuilt_rule_assets_client'; import type { IPrebuiltRuleObjectsClient } from '../rule_objects/prebuilt_rule_objects_client'; -import type { RuleVersionSpecifier } from '../../model/rule_versions/rule_version_specifier'; +import type { RuleVersionSpecifier } from './rule_version_specifier'; import { zipRuleVersions } from './zip_rule_versions'; interface GetRuleVersionsMapArgs { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_versions/rule_version_specifier.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/rule_version_specifier.ts similarity index 100% rename from x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_versions/rule_version_specifier.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_versions/rule_version_specifier.ts diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.mock.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.mock.ts index 8f9c1a6a32357..6442582c1b573 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.mock.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.mock.ts @@ -4,11 +4,24 @@ * 2.0; you may not use this file except in compliance with the Elastic License * 2.0. */ +import type { z } from '@kbn/zod'; +import type { + EqlRuleCreateFields, + QueryRuleCreateFields, + SavedQueryRuleCreateFields, + ThresholdRuleCreateFields, + ThreatMatchRuleCreateFields, + MachineLearningRuleCreateFields, + NewTermsRuleCreateFields, + EsqlRuleCreateFields, + TypeSpecificCreatePropsInternal, +} from '../../../../../../common/api/detection_engine'; +import { PrebuiltRuleAsset, type PrebuiltAssetBaseProps } from './prebuilt_rule_asset'; -import type { PrebuiltRuleAsset } from './prebuilt_rule_asset'; +type TypeSpecificCreateProps = z.infer; -export const getPrebuiltRuleMock = (rewrites?: Partial): PrebuiltRuleAsset => - ({ +export const getPrebuiltRuleMock = (rewrites?: Partial): PrebuiltRuleAsset => { + return PrebuiltRuleAsset.parse({ description: 'some description', name: 'Query with a rule id', query: 'user.name: root or user.name: admin', @@ -19,40 +32,42 @@ export const getPrebuiltRuleMock = (rewrites?: Partial): Preb rule_id: 'rule-1', version: 1, author: [], + license: 'Elastic License v2', ...rewrites, - } as PrebuiltRuleAsset); + }); +}; -export const getPrebuiltRuleWithExceptionsMock = (): PrebuiltRuleAsset => ({ - description: 'A rule with an exception list', - name: 'A rule with an exception list', - query: 'user.name: root or user.name: admin', - severity: 'high', +export const getPrebuiltQueryRuleSpecificFieldsMock = (): QueryRuleCreateFields => ({ type: 'query', - risk_score: 42, + query: 'user.name: root or user.name: admin', language: 'kuery', - rule_id: 'rule-with-exceptions', - exceptions_list: [ - { - id: 'endpoint_list', - list_id: 'endpoint_list', - namespace_type: 'agnostic', - type: 'endpoint', - }, - ], - version: 2, }); -export const getPrebuiltThreatMatchRuleMock = (): PrebuiltRuleAsset => ({ - description: 'some description', - name: 'Query with a rule id', +export const getPrebuiltEqlRuleSpecificFieldsMock = (): EqlRuleCreateFields => ({ + type: 'eql', + query: 'process where process.name == "cmd.exe"', + language: 'eql', +}); + +export const getPrebuiltSavedQueryRuleSpecificFieldsMock = (): SavedQueryRuleCreateFields => ({ + type: 'saved_query', + saved_id: 'saved-query-id', +}); + +export const getPrebuiltThresholdRuleSpecificFieldsMock = (): ThresholdRuleCreateFields => ({ + type: 'threshold', query: 'user.name: root or user.name: admin', - severity: 'high', + language: 'kuery', + threshold: { + field: 'user.name', + value: 5, + }, +}); + +export const getPrebuiltThreatMatchRuleSpecificFieldsMock = (): ThreatMatchRuleCreateFields => ({ type: 'threat_match', - risk_score: 55, + query: 'user.name: root or user.name: admin', language: 'kuery', - rule_id: 'rule-1', - version: 1, - author: [], threat_query: '*:*', threat_index: ['list-index'], threat_mapping: [ @@ -66,22 +81,115 @@ export const getPrebuiltThreatMatchRuleMock = (): PrebuiltRuleAsset => ({ ], }, ], - threat_filters: [ - { - bool: { - must: [ - { - query_string: { - query: 'host.name: linux', - analyze_wildcard: true, - time_zone: 'Zulu', - }, - }, - ], - filter: [], - should: [], - must_not: [], - }, - }, - ], + concurrent_searches: 2, + items_per_search: 10, }); + +export const getPrebuiltThreatMatchRuleMock = (): PrebuiltRuleAsset => ({ + description: 'some description', + name: 'Query with a rule id', + severity: 'high', + risk_score: 55, + rule_id: 'rule-1', + version: 1, + author: [], + license: 'Elastic License v2', + ...getPrebuiltThreatMatchRuleSpecificFieldsMock(), +}); + +export const getPrebuiltMachineLearningRuleSpecificFieldsMock = + (): MachineLearningRuleCreateFields => ({ + type: 'machine_learning', + anomaly_threshold: 50, + machine_learning_job_id: 'ml-job-id', + }); + +export const getPrebuiltNewTermsRuleSpecificFieldsMock = (): NewTermsRuleCreateFields => ({ + type: 'new_terms', + query: 'user.name: *', + language: 'kuery', + new_terms_fields: ['user.name'], + history_window_start: '1h', +}); + +export const getPrebuiltEsqlRuleSpecificFieldsMock = (): EsqlRuleCreateFields => ({ + type: 'esql', + query: 'from process where process.name == "cmd.exe"', + language: 'esql', +}); + +export const getPrebuiltRuleMockOfType = ( + type: T['type'] +): PrebuiltAssetBaseProps & + Extract & { version: number; rule_id: string } => { + let typeSpecificFields: TypeSpecificCreateProps; + + switch (type) { + case 'query': + typeSpecificFields = getPrebuiltQueryRuleSpecificFieldsMock(); + break; + case 'eql': + typeSpecificFields = getPrebuiltEqlRuleSpecificFieldsMock(); + break; + case 'saved_query': + typeSpecificFields = getPrebuiltSavedQueryRuleSpecificFieldsMock(); + break; + case 'threshold': + typeSpecificFields = getPrebuiltThresholdRuleSpecificFieldsMock(); + break; + case 'threat_match': + typeSpecificFields = getPrebuiltThreatMatchRuleSpecificFieldsMock(); + break; + case 'machine_learning': + typeSpecificFields = getPrebuiltMachineLearningRuleSpecificFieldsMock(); + break; + case 'new_terms': + typeSpecificFields = getPrebuiltNewTermsRuleSpecificFieldsMock(); + break; + case 'esql': + typeSpecificFields = getPrebuiltEsqlRuleSpecificFieldsMock(); + break; + default: + throw new Error(`Unsupported rule type: ${type}`); + } + + return { + tags: ['tag1', 'tag2'], + description: 'some description', + name: `${type} rule`, + severity: 'high', + risk_score: 55, + author: [], + license: 'Elastic License v2', + ...typeSpecificFields, + rule_id: `rule-${type}`, + version: 1, + }; +}; + +export const getPrebuiltRuleWithExceptionsMock = ( + rewrites?: Partial +): PrebuiltRuleAsset => { + const parsedFields = rewrites ? PrebuiltRuleAsset.parse(rewrites) : {}; + + return { + description: 'A rule with an exception list', + name: 'A rule with an exception list', + query: 'user.name: root or user.name: admin', + severity: 'high', + type: 'query', + risk_score: 42, + language: 'kuery', + rule_id: 'rule-with-exceptions', + exceptions_list: [ + { + id: 'endpoint_list', + list_id: 'endpoint_list', + namespace_type: 'agnostic', + type: 'endpoint', + }, + ], + version: 2, + ...parsedFields, + }; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.ts index cc7e38632547f..8069ee0385eb7 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_assets/prebuilt_rule_asset.ts @@ -20,7 +20,6 @@ function zodMaskFor() { return Object.assign({}, ...propObjects); }; } - /** * The PrebuiltRuleAsset schema is created based on the rule schema defined in our OpenAPI specs. * However, we don't need all the rule schema fields to be present in the PrebuiltRuleAsset. @@ -39,6 +38,7 @@ const BASE_PROPS_REMOVED_FROM_PREBUILT_RULE_ASSET = zodMaskFor( 'outcome', ]); +export type PrebuiltAssetBaseProps = z.infer; export const PrebuiltAssetBaseProps = BaseCreateProps.omit( BASE_PROPS_REMOVED_FROM_PREBUILT_RULE_ASSET ); @@ -65,31 +65,3 @@ export const PrebuiltRuleAsset = PrebuiltAssetBaseProps.and(TypeSpecificCreatePr version: RuleVersion, }) ); - -function createUpgradableRuleFieldsPayloadByType() { - const baseFields = Object.keys(PrebuiltAssetBaseProps.shape); - - return new Map( - TypeSpecificCreatePropsInternal.options.map((option) => { - const typeName = option.shape.type.value; - const typeSpecificFieldsForType = Object.keys(option.shape); - - return [typeName, [...baseFields, ...typeSpecificFieldsForType]]; - }) - ); -} - -/** - * Map of the fields payloads to be passed to the `upgradePrebuiltRules()` method during the - * Upgrade workflow (`/upgrade/_perform` endpoint) by type. - * - * Creating this Map dynamically, based on BaseCreateProps and TypeSpecificFields, ensures that we don't need to: - * - manually add rule types to this Map if they are created - * - manually add or remove any fields if they are added or removed to a specific rule type - * - manually add or remove any fields if we decide that they should not be part of the upgradable fields. - * - * Notice that this Map includes, for each rule type, all fields that are part of the BaseCreateProps and all fields that - * are part of the TypeSpecificFields, including those that are not part of RuleUpgradeSpecifierFields schema, where - * the user of the /upgrade/_perform endpoint can specify which fields to upgrade during the upgrade workflow. - */ -export const UPGRADABLE_FIELDS_PAYLOAD_BY_RULE_TYPE = createUpgradableRuleFieldsPayloadByType(); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_versions/get_version_buckets.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_groups/get_rule_groups.ts similarity index 75% rename from x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_versions/get_version_buckets.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_groups/get_rule_groups.ts index 0c541c0ae00ff..c9adf6db850fb 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_versions/get_version_buckets.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/model/rule_groups/get_rule_groups.ts @@ -9,7 +9,21 @@ import type { RuleResponse } from '../../../../../../common/api/detection_engine import type { RuleVersions } from '../../logic/diff/calculate_rule_diff'; import type { PrebuiltRuleAsset } from '../rule_assets/prebuilt_rule_asset'; -export interface VersionBuckets { +export interface RuleTriad { + /** + * The base version of the rule (no customizations) + */ + base?: PrebuiltRuleAsset; + /** + * The currently installed version + */ + current: RuleResponse; + /** + * The latest available version + */ + target: PrebuiltRuleAsset; +} +export interface RuleGroups { /** * Rules that are currently installed in Kibana */ @@ -21,16 +35,7 @@ export interface VersionBuckets { /** * Rules that are installed but outdated */ - upgradeableRules: Array<{ - /** - * The currently installed version - */ - current: RuleResponse; - /** - * The latest available version - */ - target: PrebuiltRuleAsset; - }>; + upgradeableRules: RuleTriad[]; /** * All available rules * (installed and not installed) @@ -38,13 +43,13 @@ export interface VersionBuckets { totalAvailableRules: PrebuiltRuleAsset[]; } -export const getVersionBuckets = (ruleVersionsMap: Map): VersionBuckets => { +export const getRuleGroups = (ruleVersionsMap: Map): RuleGroups => { const currentRules: RuleResponse[] = []; const installableRules: PrebuiltRuleAsset[] = []; const totalAvailableRules: PrebuiltRuleAsset[] = []; - const upgradeableRules: VersionBuckets['upgradeableRules'] = []; + const upgradeableRules: RuleGroups['upgradeableRules'] = []; - ruleVersionsMap.forEach(({ current, target }) => { + ruleVersionsMap.forEach(({ base, current, target }) => { if (target != null) { // If this rule is available in the package totalAvailableRules.push(target); @@ -63,6 +68,7 @@ export const getVersionBuckets = (ruleVersionsMap: Map): V if (current != null && target != null && current.version < target.version) { // If this rule is installed but outdated upgradeableRules.push({ + base, current, target, }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/mergers/apply_rule_patch.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/mergers/apply_rule_patch.ts index ba21037ba376f..becc68f3d0075 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/mergers/apply_rule_patch.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/mergers/apply_rule_patch.ts @@ -85,6 +85,8 @@ export const applyRulePatch = async ({ from: rulePatch.from ?? existingRule.from, license: rulePatch.license ?? existingRule.license, output_index: rulePatch.output_index ?? existingRule.output_index, + alias_purpose: rulePatch.alias_purpose ?? existingRule.alias_purpose, + alias_target_id: rulePatch.alias_target_id ?? existingRule.alias_target_id, timeline_id: rulePatch.timeline_id ?? existingRule.timeline_id, timeline_title: rulePatch.timeline_title ?? existingRule.timeline_title, meta: rulePatch.meta ?? existingRule.meta, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/methods/upgrade_prebuilt_rule.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/methods/upgrade_prebuilt_rule.ts index ee5686e96d130..64486bed14304 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/methods/upgrade_prebuilt_rule.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/methods/upgrade_prebuilt_rule.ts @@ -14,7 +14,7 @@ import type { PrebuiltRuleAsset } from '../../../../prebuilt_rules'; import type { IPrebuiltRuleAssetsClient } from '../../../../prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client'; import { convertAlertingRuleToRuleResponse } from '../converters/convert_alerting_rule_to_rule_response'; import { convertRuleResponseToAlertingRule } from '../converters/convert_rule_response_to_alerting_rule'; -import { applyRulePatch } from '../mergers/apply_rule_patch'; +import { applyRuleUpdate } from '../mergers/apply_rule_update'; import { ClientError, validateMlAuth } from '../utils'; import { createRule } from './create_rule'; import { getRuleByRuleId } from './get_rule_by_rule_id'; @@ -68,17 +68,17 @@ export const upgradePrebuiltRule = async ({ return createdRule; } - // Else, simply patch it. - const patchedRule = await applyRulePatch({ + // Else, recreate the rule from scratch with the passed payload. + const updatedRule = await applyRuleUpdate({ prebuiltRuleAssetClient, existingRule, - rulePatch: ruleAsset, + ruleUpdate: ruleAsset, }); - const patchedInternalRule = await rulesClient.update({ + const updatedInternalRule = await rulesClient.update({ id: existingRule.id, - data: convertRuleResponseToAlertingRule(patchedRule, actionsClient), + data: convertRuleResponseToAlertingRule(updatedRule, actionsClient), }); - return convertAlertingRuleToRuleResponse(patchedInternalRule); + return convertAlertingRuleToRuleResponse(updatedInternalRule); }; diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/update_actions.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/update_actions.ts index 40a967c068a00..93deebb4ad7d9 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/update_actions.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/update_actions.ts @@ -39,6 +39,20 @@ export default ({ getService }: FtrProviderContext) => { describe('@serverless @ess update_actions', () => { describe('updating actions', () => { + before(async () => { + await es.indices.delete({ index: 'logs-test', ignore_unavailable: true }); + await es.indices.create({ + index: 'logs-test', + mappings: { + properties: { + '@timestamp': { + type: 'date', + }, + }, + }, + }); + }); + beforeEach(async () => { await deleteAllAlerts(supertest, log, es); await deleteAllRules(supertest, log); diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts index 3c5806688cd61..03772258bd679 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts @@ -14,7 +14,7 @@ import { createRuleAssetSavedObject, createPrebuiltRuleAssetSavedObjects, installPrebuiltRules, - upgradePrebuiltRules, + performUpgradePrebuiltRules, createHistoricalPrebuiltRuleAssetSavedObjects, getPrebuiltRulesAndTimelinesStatus, installPrebuiltRulesAndTimelines, @@ -136,8 +136,11 @@ export default ({ getService }: FtrProviderContext): void => { // Increment the version of one of the installed rules and create the new rule assets ruleAssetSavedObjects[0]['security-rule'].version += 1; await createPrebuiltRuleAssetSavedObjects(es, ruleAssetSavedObjects); - // Upgrade all rules - await upgradePrebuiltRules(es, supertest); + // Upgrade all rules to target version + await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); const { stats } = await getPrebuiltRulesStatus(es, supertest); expect(stats).toMatchObject({ @@ -270,8 +273,11 @@ export default ({ getService }: FtrProviderContext): void => { createRuleAssetSavedObject({ rule_id: 'rule-1', version: 3 }), ]); - // Upgrade the rule - await upgradePrebuiltRules(es, supertest); + // Upgrade the rule to target version + await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); const { stats } = await getPrebuiltRulesStatus(es, supertest); expect(stats).toMatchObject({ diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/index.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/index.ts index 72707393c0527..46db3e2602702 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/index.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/index.ts @@ -17,6 +17,8 @@ export default ({ loadTestFile }: FtrProviderContext): void => { loadTestFile(require.resolve('./upgrade_prebuilt_rules')); loadTestFile(require.resolve('./upgrade_prebuilt_rules_with_historical_versions')); loadTestFile(require.resolve('./fleet_integration')); + loadTestFile(require.resolve('./upgrade_perform_prebuilt_rules.all_rules_mode')); + loadTestFile(require.resolve('./upgrade_perform_prebuilt_rules.specific_rules_mode')); loadTestFile(require.resolve('./upgrade_review_prebuilt_rules.rule_type_fields')); loadTestFile(require.resolve('./upgrade_review_prebuilt_rules.number_fields')); loadTestFile(require.resolve('./upgrade_review_prebuilt_rules.single_line_string_fields')); diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.all_rules_mode.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.all_rules_mode.ts new file mode 100644 index 0000000000000..2d0fe71e7d5d4 --- /dev/null +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.all_rules_mode.ts @@ -0,0 +1,490 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import expect from 'expect'; +import type SuperTest from 'supertest'; +import { cloneDeep } from 'lodash'; +import { + QueryRuleCreateFields, + EqlRuleCreateFields, + EsqlRuleCreateFields, + RuleResponse, + ThreatMatchRuleCreateFields, + ThreatMatchRule, + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION, + ModeEnum, +} from '@kbn/security-solution-plugin/common/api/detection_engine'; +import { PrebuiltRuleAsset } from '@kbn/security-solution-plugin/server/lib/detection_engine/prebuilt_rules'; +import { FtrProviderContext } from '../../../../../../ftr_provider_context'; +import { + deleteAllTimelines, + deleteAllPrebuiltRuleAssets, + createRuleAssetSavedObjectOfType, + installPrebuiltRules, + performUpgradePrebuiltRules, + patchRule, + createHistoricalPrebuiltRuleAssetSavedObjects, + reviewPrebuiltRulesToUpgrade, + getInstalledRules, + createRuleAssetSavedObject, + getWebHookAction, +} from '../../../../utils'; +import { deleteAllRules } from '../../../../../../../common/utils/security_solution'; + +export default ({ getService }: FtrProviderContext): void => { + const es = getService('es'); + const supertest = getService('supertest'); + const log = getService('log'); + const securitySolutionApi = getService('securitySolutionApi'); + + describe('@ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: ALL_RULES', () => { + beforeEach(async () => { + await deleteAllRules(supertest, log); + await deleteAllTimelines(es, log); + await deleteAllPrebuiltRuleAssets(es, log); + }); + + const CURRENT_NAME = 'My current name'; + const CURRENT_TAGS = ['current', 'tags']; + const TARGET_NAME = 'My target name'; + const TARGET_TAGS = ['target', 'tags']; + + describe(`successful updates`, () => { + const queryRule = createRuleAssetSavedObjectOfType('query'); + const eqlRule = createRuleAssetSavedObjectOfType('eql'); + const esqlRule = createRuleAssetSavedObjectOfType('esql'); + + const basePrebuiltAssets = [queryRule, eqlRule, esqlRule]; + const basePrebuiltAssetsMap = createIdToRuleMap( + basePrebuiltAssets.map((r) => r['security-rule']) + ); + + const targetPrebuiltAssets = basePrebuiltAssets.map((ruleAssetSavedObject) => { + const targetObject = cloneDeep(ruleAssetSavedObject); + targetObject['security-rule'].version += 1; + targetObject['security-rule'].name = TARGET_NAME; + targetObject['security-rule'].tags = TARGET_TAGS; + + return targetObject; + }); + + it('upgrades all upgreadeable rules fields to their BASE versions', async () => { + // Install base prebuilt detection rule + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + // Perform the upgrade, all rules' fields to their BASE versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'BASE', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(3); + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const matchingBaseAsset = basePrebuiltAssetsMap.get(updatedRule.rule_id); + if (!matchingBaseAsset) { + throw new Error(`Could not find matching base asset for rule ${updatedRule.rule_id}`); + } + + // Rule Version should be incremented by 1 + // Rule Name and Tags should match the base asset's values, not the Target asset's values + expect(updatedRule.version).toEqual(matchingBaseAsset.version + 1); + expect(updatedRule.name).toEqual(matchingBaseAsset.name); + expect(updatedRule.tags).toEqual(matchingBaseAsset.tags); + }); + + // Get installed rules + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + for (const [ruleId, installedRule] of installedRulesMap) { + const matchingBaseAsset = basePrebuiltAssetsMap.get(ruleId); + expect(installedRule.name).toEqual(matchingBaseAsset?.name); + expect(installedRule.tags).toEqual(matchingBaseAsset?.tags); + } + }); + + it('upgrades all upgreadeable rules fields to their CURRENT versions', async () => { + // Install base prebuilt detection rule + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Patch all 3 installed rules to create a current version for each + for (const baseRule of basePrebuiltAssets) { + await patchRule(supertest, log, { + rule_id: baseRule['security-rule'].rule_id, + name: CURRENT_NAME, + tags: CURRENT_TAGS, + }); + } + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + // Perform the upgrade, all rules' fields to their CURRENT versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'CURRENT', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(3); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const matchingBaseAsset = basePrebuiltAssetsMap.get(updatedRule.rule_id); + // Rule Version should be incremented by 1 + // Rule Query should match the current's version query + if (matchingBaseAsset) { + expect(updatedRule.version).toEqual(matchingBaseAsset.version + 1); + expect(updatedRule.name).toEqual(CURRENT_NAME); + expect(updatedRule.tags).toEqual(CURRENT_TAGS); + } else { + throw new Error(`Matching base asset not found for rule_id: ${updatedRule.rule_id}`); + } + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + for (const [_, installedRule] of installedRulesMap) { + expect(installedRule.name).toEqual(CURRENT_NAME); + expect(installedRule.tags).toEqual(CURRENT_TAGS); + } + }); + + it('upgrades all upgreadeable rules fields to their TARGET versions', async () => { + // Install base prebuilt detection rule + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Patch all 3 installed rules to create a current version for each + for (const baseRule of basePrebuiltAssets) { + await patchRule(supertest, log, { + rule_id: baseRule['security-rule'].rule_id, + query: CURRENT_NAME, + tags: CURRENT_TAGS, + }); + } + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + // Perform the upgrade, all rules' fields to their CURRENT versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'TARGET', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(3); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const matchingBaseAsset = basePrebuiltAssetsMap.get(updatedRule.rule_id); + + // Rule Version should be incremented by 1 + // Rule Query should match the current's version query + if (matchingBaseAsset) { + expect(updatedRule.version).toEqual(matchingBaseAsset.version + 1); + expect(updatedRule.name).toEqual(TARGET_NAME); + expect(updatedRule.tags).toEqual(TARGET_TAGS); + } else { + throw new Error(`Matching base asset not found for rule_id: ${updatedRule.rule_id}`); + } + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + for (const [_, installedRule] of installedRulesMap) { + expect(installedRule.name).toEqual(TARGET_NAME); + expect(installedRule.tags).toEqual(TARGET_TAGS); + } + }); + + it('upgrades all upgreadeable rules fields to their MERGED versions', async () => { + // Install base prebuilt detection rule + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + // Call the /upgrade/_review endpoint to save the calculated merged_versions + const reviewResponse = await reviewPrebuiltRulesToUpgrade(supertest); + const reviewRuleResponseMap = new Map( + reviewResponse.rules.map((upgradeInfo) => [ + upgradeInfo.rule_id, + { + tags: upgradeInfo.diff.fields.tags?.merged_version, + name: upgradeInfo.diff.fields.name?.merged_version, + }, + ]) + ); + + // Perform the upgrade, all rules' fields to their MERGED versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'MERGED', + }); + const updatedRulesMap = createIdToRuleMap(performUpgradeResponse.results.updated); + + // All upgrades should succeed: neither query nor tags should have a merge conflict + expect(performUpgradeResponse.summary.succeeded).toEqual(3); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + for (const [ruleId, installedRule] of installedRulesMap) { + expect(installedRule.name).toEqual(updatedRulesMap.get(ruleId)?.name); + expect(installedRule.name).toEqual(reviewRuleResponseMap.get(ruleId)?.name); + expect(installedRule.tags).toEqual(updatedRulesMap.get(ruleId)?.tags); + expect(installedRule.tags).toEqual(reviewRuleResponseMap.get(ruleId)?.tags); + } + }); + }); + + describe('edge cases and unhappy paths', () => { + const firstQueryRule = createRuleAssetSavedObject({ + type: 'query', + language: 'kuery', + rule_id: 'query-rule-1', + }); + const secondQueryRule = createRuleAssetSavedObject({ + type: 'query', + language: 'kuery', + rule_id: 'query-rule-2', + }); + const eqlRule = createRuleAssetSavedObject({ + type: 'eql', + language: 'eql', + rule_id: 'eql-rule', + }); + + const basePrebuiltAssets = [firstQueryRule, eqlRule, secondQueryRule]; + + it('rejects all updates of rules which have a rule type change if the pick_version is not TARGET', async () => { + // Install base prebuilt detection rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Mock a rule type change to 'ml' to the first two rules of the basePrebuiltAssets array + const targetMLPrebuiltAssets = basePrebuiltAssets + .slice(0, 2) + .map((ruleAssetSavedObject) => { + const targetObject = cloneDeep(ruleAssetSavedObject); + + return { + ...targetObject, + ...createRuleAssetSavedObject({ + rule_id: targetObject['security-rule'].rule_id, + version: targetObject['security-rule'].version + 1, + type: 'machine_learning', + machine_learning_job_id: 'job_id', + anomaly_threshold: 1, + }), + }; + }); + + // Mock an normal update of the rule 'query-rule-2', with NO rule type change + const targetAssetSameTypeUpdate = createRuleAssetSavedObject({ + type: 'query', + language: 'kuery', + rule_id: 'query-rule-2', + version: 2, + }); + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [ + ...targetMLPrebuiltAssets, + targetAssetSameTypeUpdate, + ]); + + // Perform the upgrade, all rules' fields to their BASE versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'BASE', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); // update of same type + expect(performUpgradeResponse.summary.failed).toEqual(2); // updates with rule type change + + expect(performUpgradeResponse.errors).toHaveLength(2); + performUpgradeResponse.errors.forEach((error) => { + const ruleId = error.rules[0].rule_id; + expect(error.message).toContain( + `Rule update for rule ${ruleId} has a rule type change. All 'pick_version' values for rule must match 'TARGET'` + ); + }); + }); + + it('rejects updates of rules with a pick_version of MERGED which have fields which result in conflicts in the three way diff calculations', async () => { + // Install base prebuilt detection rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + // Patch all 3 installed rules to create a current version for each + for (const baseRule of basePrebuiltAssets) { + await patchRule(supertest, log, { + rule_id: baseRule['security-rule'].rule_id, + name: CURRENT_NAME, + tags: CURRENT_TAGS, + }); + } + + const targetPrebuiltAssets = basePrebuiltAssets.map((ruleAssetSavedObject) => { + const targetObject = cloneDeep(ruleAssetSavedObject); + targetObject['security-rule'].version += 1; + targetObject['security-rule'].name = TARGET_NAME; + targetObject['security-rule'].tags = TARGET_TAGS; + + return targetObject; + }); + + // Create new versions of the assets of the installed rules + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + // Perform the upgrade, all rules' fields to their MERGED versions + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'MERGED', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(0); // all rules have conflicts + expect(performUpgradeResponse.summary.failed).toEqual(3); // all rules have conflicts + + performUpgradeResponse.errors.forEach((error) => { + const ruleId = error.rules[0].rule_id; + expect(error.message).toContain( + `Merge conflicts found in rule '${ruleId}' for fields: name, tags. Please resolve the conflict manually or choose another value for 'pick_version'` + ); + }); + }); + + it('preserves FIELDS_TO_UPGRADE_TO_CURRENT_VERSION when upgrading to TARGET version with undefined fields', async () => { + const baseRule = + createRuleAssetSavedObjectOfType('threat_match'); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [baseRule]); + await installPrebuiltRules(es, supertest); + + const ruleId = baseRule['security-rule'].rule_id; + + const installedBaseRule = ( + await securitySolutionApi.readRule({ + query: { + rule_id: ruleId, + }, + }) + ).body as ThreatMatchRule; + + // Patch the installed rule to set all FIELDS_TO_UPGRADE_TO_CURRENT_VERSION to some defined value + const currentValues: { [key: string]: unknown } = { + enabled: true, + exceptions_list: [ + { + id: 'test-list', + list_id: 'test-list', + type: 'detection', + namespace_type: 'single', + } as const, + ], + alert_suppression: { + group_by: ['host.name'], + duration: { value: 5, unit: 'm' as const }, + }, + actions: [await createAction(supertest)], + response_actions: [ + { + params: { + command: 'isolate' as const, + comment: 'comment', + }, + action_type_id: '.endpoint' as const, + }, + ], + meta: { some_key: 'some_value' }, + output_index: '.siem-signals-default', + namespace: 'default', + concurrent_searches: 5, + items_per_search: 100, + }; + + await securitySolutionApi.updateRule({ + body: { + ...installedBaseRule, + ...currentValues, + id: undefined, + }, + }); + + // Create a target version with undefined values for these fields + const targetRule = cloneDeep(baseRule); + targetRule['security-rule'].version += 1; + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + // @ts-expect-error + targetRule['security-rule'][field] = undefined; + }); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetRule]); + + // Perform the upgrade + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.ALL_RULES, + pick_version: 'TARGET', + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); + const upgradedRule = performUpgradeResponse.results.updated[0] as ThreatMatchRule; + + // Check that all FIELDS_TO_UPGRADE_TO_CURRENT_VERSION still have their "current" values + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(upgradedRule[field]).toEqual(currentValues[field]); + }); + + // Verify the installed rule + const installedRules = await getInstalledRules(supertest); + const installedRule = installedRules.data.find( + (rule) => rule.rule_id === baseRule['security-rule'].rule_id + ) as ThreatMatchRule; + + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(installedRule[field]).toEqual(currentValues[field]); + }); + }); + }); + }); +}; + +function createIdToRuleMap(rules: Array) { + return new Map(rules.map((rule) => [rule.rule_id, rule])); +} + +async function createAction(supertest: SuperTest.Agent) { + const createConnector = async (payload: Record) => + (await supertest.post('/api/actions/action').set('kbn-xsrf', 'true').send(payload).expect(200)) + .body; + + const createWebHookConnector = () => createConnector(getWebHookAction()); + + const webHookAction = await createWebHookConnector(); + + const defaultRuleAction = { + id: webHookAction.id, + action_type_id: '.webhook' as const, + group: 'default' as const, + params: { + body: '{"test":"a default action"}', + }, + frequency: { + notifyWhen: 'onThrottleInterval' as const, + summary: true, + throttle: '1h' as const, + }, + uuid: 'd487ec3d-05f2-44ad-8a68-11c97dc92202', + }; + + return defaultRuleAction; +} diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.specific_rules_mode.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.specific_rules_mode.ts new file mode 100644 index 0000000000000..8c086c46927e7 --- /dev/null +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_perform_prebuilt_rules.specific_rules_mode.ts @@ -0,0 +1,861 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import expect from 'expect'; +import type SuperTest from 'supertest'; +import { cloneDeep } from 'lodash'; +import { + QueryRuleCreateFields, + EqlRuleCreateFields, + EsqlRuleCreateFields, + ThreatMatchRuleCreateFields, + RuleResponse, + ModeEnum, + PickVersionValues, + RuleEqlQuery, + EqlRule, + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION, +} from '@kbn/security-solution-plugin/common/api/detection_engine'; +import { PrebuiltRuleAsset } from '@kbn/security-solution-plugin/server/lib/detection_engine/prebuilt_rules'; +import { ThreatMatchRule } from '@kbn/security-solution-plugin/common/api/detection_engine/model/rule_schema/rule_schemas.gen'; +import { FtrProviderContext } from '../../../../../../ftr_provider_context'; +import { + deleteAllTimelines, + deleteAllPrebuiltRuleAssets, + createRuleAssetSavedObjectOfType, + installPrebuiltRules, + performUpgradePrebuiltRules, + patchRule, + getInstalledRules, + createHistoricalPrebuiltRuleAssetSavedObjects, + reviewPrebuiltRulesToUpgrade, + createRuleAssetSavedObject, + getWebHookAction, +} from '../../../../utils'; +import { deleteAllRules } from '../../../../../../../common/utils/security_solution'; + +export default ({ getService }: FtrProviderContext): void => { + const es = getService('es'); + const supertest = getService('supertest'); + const log = getService('log'); + const securitySolutionApi = getService('securitySolutionApi'); + + describe('@ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: SPECIFIC_RULES', () => { + beforeEach(async () => { + await deleteAllRules(supertest, log); + await deleteAllTimelines(es, log); + await deleteAllPrebuiltRuleAssets(es, log); + }); + + const CURRENT_NAME = 'My current name'; + const CURRENT_TAGS = ['current', 'tags']; + const TARGET_NAME = 'My target name'; + const TARGET_TAGS = ['target', 'tags']; + + describe('successful updates', () => { + const queryRule = createRuleAssetSavedObjectOfType('query'); + const eqlRule = createRuleAssetSavedObjectOfType('eql'); + const esqlRule = createRuleAssetSavedObjectOfType('esql'); + + const basePrebuiltAssets = [queryRule, eqlRule, esqlRule]; + + const targetPrebuiltAssets = basePrebuiltAssets.map((ruleAssetSavedObject) => { + const targetObject = cloneDeep(ruleAssetSavedObject); + targetObject['security-rule'].version += 1; + targetObject['security-rule'].name = TARGET_NAME; + targetObject['security-rule'].tags = TARGET_TAGS; + return targetObject; + }); + + it('upgrades specific rules to their BASE versions', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const rulesToUpgrade = basePrebuiltAssets.map((rule) => ({ + rule_id: rule['security-rule'].rule_id, + revision: 0, + version: rule['security-rule'].version + 1, + })); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'BASE', + rules: rulesToUpgrade, + }); + + const expectedResults = basePrebuiltAssets.map((asset) => ({ + rule_id: asset['security-rule'].rule_id, + version: asset['security-rule'].version + 1, + name: asset['security-rule'].name, + tags: asset['security-rule'].tags, + })); + + expect(performUpgradeResponse.summary.succeeded).toEqual(basePrebuiltAssets.length); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const expected = expectedResults.find((r) => r.rule_id === updatedRule.rule_id); + expect(updatedRule.version).toEqual(expected?.version); + expect(updatedRule.name).toEqual(expected?.name); + expect(updatedRule.tags).toEqual(expected?.tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + expectedResults.forEach((expected) => { + const installedRule = installedRulesMap.get(expected.rule_id); + expect(installedRule?.name).toEqual(expected.name); + expect(installedRule?.tags).toEqual(expected.tags); + }); + }); + + it('upgrades specific rules to their CURRENT versions', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + for (const baseRule of basePrebuiltAssets) { + await patchRule(supertest, log, { + rule_id: baseRule['security-rule'].rule_id, + name: CURRENT_NAME, + tags: CURRENT_TAGS, + }); + } + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const rulesToUpgrade = basePrebuiltAssets.map((rule) => ({ + rule_id: rule['security-rule'].rule_id, + revision: 1, + version: rule['security-rule'].version + 1, + })); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'CURRENT', + rules: rulesToUpgrade, + }); + + const expectedResults = basePrebuiltAssets.map((asset) => ({ + rule_id: asset['security-rule'].rule_id, + name: CURRENT_NAME, + tags: CURRENT_TAGS, + })); + + expect(performUpgradeResponse.summary.succeeded).toEqual(basePrebuiltAssets.length); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const expected = expectedResults.find((r) => r.rule_id === updatedRule.rule_id); + expect(updatedRule.name).toEqual(expected?.name); + expect(updatedRule.tags).toEqual(expected?.tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + expectedResults.forEach((expected) => { + const installedRule = installedRulesMap.get(expected.rule_id); + expect(installedRule?.name).toEqual(expected.name); + expect(installedRule?.tags).toEqual(expected.tags); + }); + }); + + it('upgrades specific rules to their TARGET versions', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const rulesToUpgrade = basePrebuiltAssets.map((rule) => ({ + rule_id: rule['security-rule'].rule_id, + revision: 0, + version: rule['security-rule'].version + 1, + })); + + const expectedResults = basePrebuiltAssets.map((asset) => ({ + rule_id: asset['security-rule'].rule_id, + name: TARGET_NAME, + tags: TARGET_TAGS, + })); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: rulesToUpgrade, + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(basePrebuiltAssets.length); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const expected = expectedResults.find((r) => r.rule_id === updatedRule.rule_id); + expect(updatedRule.name).toEqual(expected?.name); + expect(updatedRule.tags).toEqual(expected?.tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + expectedResults.forEach((expected) => { + const installedRule = installedRulesMap.get(expected.rule_id); + expect(installedRule?.name).toEqual(expected.name); + expect(installedRule?.tags).toEqual(expected.tags); + }); + }); + + it('upgrades specific rules to their MERGED versions', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const reviewResponse = await reviewPrebuiltRulesToUpgrade(supertest); + const expectedResults = reviewResponse.rules.map((upgradeInfo) => ({ + rule_id: upgradeInfo.rule_id, + name: upgradeInfo.diff.fields.name?.merged_version, + tags: upgradeInfo.diff.fields.tags?.merged_version, + })); + + const rulesToUpgrade = basePrebuiltAssets.map((rule) => ({ + rule_id: rule['security-rule'].rule_id, + revision: 0, + version: rule['security-rule'].version + 1, + })); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'MERGED', + rules: rulesToUpgrade, + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(basePrebuiltAssets.length); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const expected = expectedResults.find((r) => r.rule_id === updatedRule.rule_id); + expect(updatedRule.name).toEqual(expected?.name); + expect(updatedRule.tags).toEqual(expected?.tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + expectedResults.forEach((expected) => { + const installedRule = installedRulesMap.get(expected.rule_id); + expect(installedRule?.name).toEqual(expected.name); + expect(installedRule?.tags).toEqual(expected.tags); + }); + }); + + it('upgrades specific rules to their TARGET versions but overrides some fields with `fields` in the request payload', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const rulesToUpgrade = basePrebuiltAssets.map((rule) => ({ + rule_id: rule['security-rule'].rule_id, + revision: 0, + version: rule['security-rule'].version + 1, + fields: { + name: { pick_version: 'BASE' as PickVersionValues }, + }, + })); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: rulesToUpgrade, + }); + + const expectedResults = basePrebuiltAssets.map((asset) => ({ + rule_id: asset['security-rule'].rule_id, + name: asset['security-rule'].name, + tags: TARGET_TAGS, + })); + + expect(performUpgradeResponse.summary.succeeded).toEqual(basePrebuiltAssets.length); + + performUpgradeResponse.results.updated.forEach((updatedRule) => { + const expected = expectedResults.find((r) => r.rule_id === updatedRule.rule_id); + expect(updatedRule.name).toEqual(expected?.name); + expect(updatedRule.tags).toEqual(expected?.tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + expectedResults.forEach((expected) => { + const installedRule = installedRulesMap.get(expected.rule_id); + expect(installedRule?.name).toEqual(expected.name); + expect(installedRule?.tags).toEqual(expected.tags); + }); + }); + + it('upgrades specific rules with different pick_version at global, rule, and field levels', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + for (const baseRule of basePrebuiltAssets) { + await patchRule(supertest, log, { + rule_id: baseRule['security-rule'].rule_id, + name: CURRENT_NAME, + tags: CURRENT_TAGS, + }); + } + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, targetPrebuiltAssets); + + const rulesToUpgrade = [ + { + rule_id: basePrebuiltAssets[0]['security-rule'].rule_id, + revision: 1, + version: basePrebuiltAssets[0]['security-rule'].version + 1, + pick_version: 'CURRENT' as PickVersionValues, + }, + { + rule_id: basePrebuiltAssets[1]['security-rule'].rule_id, + revision: 1, + version: basePrebuiltAssets[1]['security-rule'].version + 1, + fields: { + name: { pick_version: 'TARGET' as PickVersionValues }, + tags: { pick_version: 'BASE' as PickVersionValues }, + }, + }, + { + rule_id: basePrebuiltAssets[2]['security-rule'].rule_id, + revision: 1, + version: basePrebuiltAssets[2]['security-rule'].version + 1, + }, + ]; + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'BASE', + rules: rulesToUpgrade, + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(3); + const updatedRulesMap = createIdToRuleMap(performUpgradeResponse.results.updated); + + const expectedResults = [ + { name: CURRENT_NAME, tags: CURRENT_TAGS }, + { name: TARGET_NAME, tags: basePrebuiltAssets[1]['security-rule'].tags }, + { + name: basePrebuiltAssets[2]['security-rule'].name, + tags: basePrebuiltAssets[2]['security-rule'].tags, + }, + ]; + + basePrebuiltAssets.forEach((asset, index) => { + const ruleId = asset['security-rule'].rule_id; + const updatedRule = updatedRulesMap.get(ruleId); + expect(updatedRule?.name).toEqual(expectedResults[index].name); + expect(updatedRule?.tags).toEqual(expectedResults[index].tags); + }); + + const installedRules = await getInstalledRules(supertest); + const installedRulesMap = createIdToRuleMap(installedRules.data); + + basePrebuiltAssets.forEach((asset, index) => { + const ruleId = asset['security-rule'].rule_id; + const installedRule = installedRulesMap.get(ruleId); + expect(installedRule?.name).toEqual(expectedResults[index].name); + expect(installedRule?.tags).toEqual(expectedResults[index].tags); + }); + }); + + it('successfully resolves a non-resolvable conflict by using pick_version:RESOLVED for that field', async () => { + const baseEqlRule = createRuleAssetSavedObjectOfType('eql'); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [baseEqlRule]); + await installPrebuiltRules(es, supertest); + + // Patch the installed rule to edit its query + const patchedQuery = 'sequence by process.name [MY CURRENT QUERY]'; + await patchRule(supertest, log, { + rule_id: baseEqlRule['security-rule'].rule_id, + query: patchedQuery, + }); + + // Create a new version of the prebuilt rule asset with a different query and generate the conflict + const targetEqlRule = cloneDeep(baseEqlRule); + targetEqlRule['security-rule'].version += 1; + targetEqlRule['security-rule'].query = 'sequence by process.name [MY TARGET QUERY]'; + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetEqlRule]); + + const resolvedValue = { + query: 'sequence by process.name [MY RESOLVED QUERY]', + language: 'eql', + filters: [], + }; + + // Perform the upgrade with manual conflict resolution + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'MERGED', + rules: [ + { + rule_id: baseEqlRule['security-rule'].rule_id, + revision: 1, + version: baseEqlRule['security-rule'].version + 1, + fields: { + eql_query: { + pick_version: 'RESOLVED', + resolved_value: resolvedValue as RuleEqlQuery, + }, + }, + }, + ], + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); + const updatedRule = performUpgradeResponse.results.updated[0] as EqlRule; + expect(updatedRule.rule_id).toEqual(baseEqlRule['security-rule'].rule_id); + expect(updatedRule.query).toEqual(resolvedValue.query); + expect(updatedRule.filters).toEqual(resolvedValue.filters); + expect(updatedRule.language).toEqual(resolvedValue.language); + + const installedRules = await getInstalledRules(supertest); + const installedRule = installedRules.data.find( + (rule) => rule.rule_id === baseEqlRule['security-rule'].rule_id + ) as EqlRule; + expect(installedRule?.query).toEqual(resolvedValue.query); + expect(installedRule?.filters).toEqual(resolvedValue.filters); + expect(installedRule?.language).toEqual(resolvedValue.language); + }); + }); + + describe('edge cases and unhappy paths', () => { + const queryRule = createRuleAssetSavedObject({ + type: 'query', + language: 'kuery', + rule_id: 'query-rule', + }); + const eqlRule = createRuleAssetSavedObject({ + type: 'eql', + language: 'eql', + rule_id: 'eql-rule', + }); + + const basePrebuiltAssets = [queryRule, eqlRule]; + + it('rejects updates when rule type changes and pick_version is not TARGET at all levels', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + const targetMLRule = createRuleAssetSavedObject({ + rule_id: queryRule['security-rule'].rule_id, + version: queryRule['security-rule'].version + 1, + type: 'machine_learning', + machine_learning_job_id: 'job_id', + anomaly_threshold: 1, + }); + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetMLRule]); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'BASE', + rules: [ + { + rule_id: queryRule['security-rule'].rule_id, + revision: 0, + version: queryRule['security-rule'].version + 1, + }, + ], + }); + + expect(performUpgradeResponse.summary.failed).toEqual(1); + expect(performUpgradeResponse.errors[0].message).toContain( + 'Rule update for rule query-rule has a rule type change' + ); + }); + + it('rejects updates when incompatible fields are provided for a rule type', async () => { + const baseEqlRule = createRuleAssetSavedObjectOfType('eql'); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [baseEqlRule]); + await installPrebuiltRules(es, supertest); + + // Create a new version of the prebuilt rule asset with a different query and generate the conflict + const targetEqlRule = cloneDeep(baseEqlRule); + targetEqlRule['security-rule'].version += 1; + targetEqlRule['security-rule'].query = 'sequence by process.name [MY TARGET QUERY]'; + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetEqlRule]); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: [ + { + rule_id: baseEqlRule['security-rule'].rule_id, + revision: 0, + version: baseEqlRule['security-rule'].version + 1, + fields: { + machine_learning_job_id: { pick_version: 'TARGET' }, + }, + }, + ], + }); + + expect(performUpgradeResponse.summary.failed).toEqual(1); + expect(performUpgradeResponse.errors[0].message).toContain( + "machine_learning_job_id is not a valid upgradeable field for type 'eql'" + ); + }); + + it('rejects updates with NON_SOLVABLE conflicts when using MERGED pick_version', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + await patchRule(supertest, log, { + rule_id: queryRule['security-rule'].rule_id, + name: CURRENT_NAME, + }); + + const targetQueryRule = cloneDeep(queryRule); + targetQueryRule['security-rule'].version += 1; + targetQueryRule['security-rule'].name = TARGET_NAME; + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetQueryRule]); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'MERGED', + rules: [ + { + rule_id: queryRule['security-rule'].rule_id, + revision: 1, + version: queryRule['security-rule'].version + 1, + }, + ], + }); + + expect(performUpgradeResponse.summary.failed).toEqual(1); + expect(performUpgradeResponse.errors[0].message).toContain( + `Automatic merge calculation for field 'name' in rule of rule_id ${performUpgradeResponse.errors[0].rules[0].rule_id} resulted in a conflict. Please resolve the conflict manually or choose another value for 'pick_version'` + ); + }); + + it('allows updates with NON_SOLVABLE conflicts when specific fields have non-MERGED pick_version', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + await patchRule(supertest, log, { + rule_id: queryRule['security-rule'].rule_id, + name: CURRENT_NAME, + }); + + const targetQueryRule = cloneDeep(queryRule); + targetQueryRule['security-rule'].version += 1; + targetQueryRule['security-rule'].name = TARGET_NAME; + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetQueryRule]); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'MERGED', + rules: [ + { + rule_id: queryRule['security-rule'].rule_id, + revision: 1, + version: queryRule['security-rule'].version + 1, + fields: { + name: { pick_version: 'TARGET' }, + }, + }, + ], + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); + expect(performUpgradeResponse.results.updated[0].name).toEqual(TARGET_NAME); + + const installedRules = await getInstalledRules(supertest); + const installedRule = installedRules.data.find( + (rule) => rule.rule_id === queryRule['security-rule'].rule_id + ); + expect(installedRule?.name).toEqual(TARGET_NAME); + }); + + it('rejects updates for specific fields with MERGED pick_version and NON_SOLVABLE conflicts', async () => { + await createHistoricalPrebuiltRuleAssetSavedObjects(es, basePrebuiltAssets); + await installPrebuiltRules(es, supertest); + + await patchRule(supertest, log, { + rule_id: queryRule['security-rule'].rule_id, + name: CURRENT_NAME, + }); + + const targetQueryRule = cloneDeep(queryRule); + targetQueryRule['security-rule'].version += 1; + targetQueryRule['security-rule'].name = TARGET_NAME; + + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetQueryRule]); + + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: [ + { + rule_id: queryRule['security-rule'].rule_id, + revision: 1, + version: queryRule['security-rule'].version + 1, + fields: { + name: { pick_version: 'MERGED' }, + }, + }, + ], + }); + + expect(performUpgradeResponse.summary.failed).toEqual(1); + expect(performUpgradeResponse.errors[0].message).toContain( + `Automatic merge calculation for field 'name' in rule of rule_id ${performUpgradeResponse.errors[0].rules[0].rule_id} resulted in a conflict. Please resolve the conflict manually or choose another value for 'pick_version'.` + ); + }); + + it('preserves FIELDS_TO_UPGRADE_TO_CURRENT_VERSION when upgrading to TARGET version with undefined fields', async () => { + const baseRule = + createRuleAssetSavedObjectOfType('threat_match'); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [baseRule]); + await installPrebuiltRules(es, supertest); + + const ruleId = baseRule['security-rule'].rule_id; + + const installedBaseRule = ( + await securitySolutionApi.readRule({ + query: { + rule_id: ruleId, + }, + }) + ).body as ThreatMatchRule; + + // Patch the installed rule to set all FIELDS_TO_UPGRADE_TO_CURRENT_VERSION to some defined value + const currentValues: { [key: string]: unknown } = { + enabled: true, + exceptions_list: [ + { + id: 'test-list', + list_id: 'test-list', + type: 'detection', + namespace_type: 'single', + } as const, + ], + alert_suppression: { + group_by: ['host.name'], + duration: { value: 5, unit: 'm' as const }, + }, + actions: [await createAction(supertest)], + response_actions: [ + { + params: { + command: 'isolate' as const, + comment: 'comment', + }, + action_type_id: '.endpoint' as const, + }, + ], + meta: { some_key: 'some_value' }, + output_index: '.siem-signals-default', + namespace: 'default', + concurrent_searches: 5, + items_per_search: 100, + }; + + await securitySolutionApi.updateRule({ + body: { + ...installedBaseRule, + ...currentValues, + id: undefined, + }, + }); + + // Create a target version with undefined values for these fields + const targetRule = cloneDeep(baseRule); + targetRule['security-rule'].version += 1; + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + // @ts-expect-error + targetRule['security-rule'][field] = undefined; + }); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetRule]); + + // Perform the upgrade + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: [ + { + rule_id: baseRule['security-rule'].rule_id, + revision: 1, + version: baseRule['security-rule'].version + 1, + }, + ], + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); + const upgradedRule = performUpgradeResponse.results.updated[0] as ThreatMatchRule; + + // Check that all FIELDS_TO_UPGRADE_TO_CURRENT_VERSION still have their "current" values + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(upgradedRule[field]).toEqual(currentValues[field]); + }); + + // Verify the installed rule + const installedRules = await getInstalledRules(supertest); + const installedRule = installedRules.data.find( + (rule) => rule.rule_id === baseRule['security-rule'].rule_id + ) as ThreatMatchRule; + + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(installedRule[field]).toEqual(currentValues[field]); + }); + }); + + it('preserves FIELDS_TO_UPGRADE_TO_CURRENT_VERSION when fields are attempted to be updated via resolved values', async () => { + const baseRule = + createRuleAssetSavedObjectOfType('threat_match'); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [baseRule]); + await installPrebuiltRules(es, supertest); + + const ruleId = baseRule['security-rule'].rule_id; + + const installedBaseRule = ( + await securitySolutionApi.readRule({ + query: { + rule_id: ruleId, + }, + }) + ).body as ThreatMatchRule; + + // Set current values for FIELDS_TO_UPGRADE_TO_CURRENT_VERSION + const currentValues: { [key: string]: unknown } = { + enabled: true, + exceptions_list: [ + { + id: 'test-list', + list_id: 'test-list', + type: 'detection', + namespace_type: 'single', + } as const, + ], + alert_suppression: { + group_by: ['host.name'], + duration: { value: 5, unit: 'm' as const }, + }, + actions: [await createAction(supertest)], + response_actions: [ + { + params: { + command: 'isolate' as const, + comment: 'comment', + }, + action_type_id: '.endpoint' as const, + }, + ], + meta: { some_key: 'some_value' }, + output_index: '.siem-signals-default', + namespace: 'default', + concurrent_searches: 5, + items_per_search: 100, + }; + + await securitySolutionApi.updateRule({ + body: { + ...installedBaseRule, + ...currentValues, + id: undefined, + }, + }); + + // Create a target version with undefined values for these fields + const targetRule = cloneDeep(baseRule); + targetRule['security-rule'].version += 1; + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + // @ts-expect-error + targetRule['security-rule'][field] = undefined; + }); + await createHistoricalPrebuiltRuleAssetSavedObjects(es, [targetRule]); + + // Create resolved values different from current values + const resolvedValues: { [key: string]: unknown } = { + exceptions_list: [], + alert_suppression: { + group_by: ['test'], + duration: { value: 10, unit: 'm' as const }, + }, + }; + + const fields = Object.fromEntries( + Object.keys(resolvedValues).map((field) => [ + field, + { + pick_version: 'RESOLVED' as PickVersionValues, + resolved_value: resolvedValues[field], + }, + ]) + ); + + // Perform the upgrade with resolved values + const performUpgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: ModeEnum.SPECIFIC_RULES, + pick_version: 'TARGET', + rules: [ + { + rule_id: baseRule['security-rule'].rule_id, + revision: 1, + version: baseRule['security-rule'].version + 1, + fields, + }, + ], + }); + + expect(performUpgradeResponse.summary.succeeded).toEqual(1); + const upgradedRule = performUpgradeResponse.results.updated[0] as ThreatMatchRule; + + // Check that all FIELDS_TO_UPGRADE_TO_CURRENT_VERSION still have their "current" values + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(upgradedRule[field]).toEqual(currentValues[field]); + }); + + // Verify the installed rule + const installedRules = await getInstalledRules(supertest); + const installedRule = installedRules.data.find( + (rule) => rule.rule_id === baseRule['security-rule'].rule_id + ) as ThreatMatchRule; + + FIELDS_TO_UPGRADE_TO_CURRENT_VERSION.forEach((field) => { + expect(installedRule[field]).toEqual(currentValues[field]); + }); + }); + }); + }); +}; + +function createIdToRuleMap(rules: Array) { + return new Map(rules.map((rule) => [rule.rule_id, rule])); +} + +async function createAction(supertest: SuperTest.Agent) { + const createConnector = async (payload: Record) => + (await supertest.post('/api/actions/action').set('kbn-xsrf', 'true').send(payload).expect(200)) + .body; + + const createWebHookConnector = () => createConnector(getWebHookAction()); + + const webHookAction = await createWebHookConnector(); + + const defaultRuleAction = { + id: webHookAction.id, + action_type_id: '.webhook' as const, + group: 'default' as const, + params: { + body: '{"test":"a default action"}', + }, + frequency: { + notifyWhen: 'onThrottleInterval' as const, + summary: true, + throttle: '1h' as const, + }, + uuid: 'd487ec3d-05f2-44ad-8a68-11c97dc92202', + }; + + return defaultRuleAction; +} diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules.ts index cd336f91fae13..a23ddf40979f6 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules.ts @@ -16,7 +16,7 @@ import { getPrebuiltRulesAndTimelinesStatus, getPrebuiltRulesStatus, installPrebuiltRules, - upgradePrebuiltRules, + performUpgradePrebuiltRules, fetchRule, patchRule, } from '../../../../utils'; @@ -100,7 +100,10 @@ export default ({ getService }: FtrProviderContext): void => { expect(statusResponse.stats.num_prebuilt_rules_to_upgrade).toBe(1); // Call the install prebuilt rules again and check that the outdated rule was updated - const response = await upgradePrebuiltRules(es, supertest); + const response = await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(response.summary.succeeded).toBe(1); expect(response.summary.skipped).toBe(0); }); @@ -121,7 +124,10 @@ export default ({ getService }: FtrProviderContext): void => { expect(installResponse.summary.skipped).toBe(0); // Call the upgrade prebuilt rules endpoint and check that no rules were updated - const upgradeResponse = await upgradePrebuiltRules(es, supertest); + const upgradeResponse = await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(upgradeResponse.summary.succeeded).toBe(0); expect(upgradeResponse.summary.skipped).toBe(0); }); @@ -178,7 +184,10 @@ export default ({ getService }: FtrProviderContext): void => { ]); // Upgrade to a newer version with the same type - await upgradePrebuiltRules(es, supertest); + await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(await fetchRule(supertest, { ruleId: 'rule-to-test-1' })).toMatchObject({ id: initialRuleSoId, @@ -186,8 +195,7 @@ export default ({ getService }: FtrProviderContext): void => { enabled: false, actions, exceptions_list: exceptionsList, - timeline_id: 'some-timeline-id', - timeline_title: 'Some timeline title', + // current values for timeline_id and timeline_title are lost when updating to TARGET version }); }); }); @@ -250,7 +258,10 @@ export default ({ getService }: FtrProviderContext): void => { ]); // Upgrade to a newer version with a different rule type - await upgradePrebuiltRules(es, supertest); + await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(await fetchRule(supertest, { ruleId: 'rule-to-test-2' })).toMatchObject({ id: initialRuleSoId, diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules_with_historical_versions.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules_with_historical_versions.ts index 049ae3a5a6fd8..0eb37b1112f27 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules_with_historical_versions.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules_with_historical_versions.ts @@ -15,7 +15,7 @@ import { createHistoricalPrebuiltRuleAssetSavedObjects, getPrebuiltRulesStatus, installPrebuiltRules, - upgradePrebuiltRules, + performUpgradePrebuiltRules, } from '../../../../utils'; import { deleteAllRules } from '../../../../../../../common/utils/security_solution'; @@ -110,7 +110,10 @@ export default ({ getService }: FtrProviderContext): void => { expect(statusResponse.stats.num_prebuilt_rules_to_upgrade).toBe(1); // Call the upgrade prebuilt rules endpoint and check that the outdated rule was updated - const response = await upgradePrebuiltRules(es, supertest); + const response = await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(response.summary.succeeded).toBe(1); expect(response.summary.total).toBe(1); @@ -138,7 +141,10 @@ export default ({ getService }: FtrProviderContext): void => { expect(statusResponse.stats.num_prebuilt_rules_to_install).toBe(0); // Call the upgrade prebuilt rules endpoint and check that the outdated rule was updated - const response = await upgradePrebuiltRules(es, supertest); + const response = await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(response.summary.succeeded).toBe(1); expect(response.summary.total).toBe(1); diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/update_prebuilt_rules_package/trial_license_complete_tier/update_prebuilt_rules_package.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/update_prebuilt_rules_package/trial_license_complete_tier/update_prebuilt_rules_package.ts index 8e26b089a9f80..b551d793406ce 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/update_prebuilt_rules_package/trial_license_complete_tier/update_prebuilt_rules_package.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/update_prebuilt_rules_package/trial_license_complete_tier/update_prebuilt_rules_package.ts @@ -19,7 +19,7 @@ import { getPrebuiltRulesStatus, installPrebuiltRules, installPrebuiltRulesPackageByVersion, - upgradePrebuiltRules, + performUpgradePrebuiltRules, reviewPrebuiltRulesToInstall, reviewPrebuiltRulesToUpgrade, } from '../../../../utils'; @@ -227,12 +227,13 @@ export default ({ getService }: FtrProviderContext): void => { prebuiltRulesToUpgradeReviewAfterLatestPackageInstallation.stats.num_rules_to_upgrade_total ).toBe(statusAfterLatestPackageInstallation.stats.num_prebuilt_rules_to_upgrade); - // Call the upgrade _perform endpoint and verify that the number of upgraded rules is the same as the one - // returned by the _review endpoint and the status endpoint - const upgradePrebuiltRulesResponseAfterLatestPackageInstallation = await upgradePrebuiltRules( - es, - supertest - ); + // Call the upgrade _perform endpoint to upgrade all rules to their target version and verify that the number + // of upgraded rules is the same as the one returned by the _review endpoint and the status endpoint + const upgradePrebuiltRulesResponseAfterLatestPackageInstallation = + await performUpgradePrebuiltRules(es, supertest, { + mode: 'ALL_RULES', + pick_version: 'TARGET', + }); expect(upgradePrebuiltRulesResponseAfterLatestPackageInstallation.summary.succeeded).toEqual( statusAfterLatestPackageInstallation.stats.num_prebuilt_rules_to_upgrade diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/export_rules.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/export_rules.ts index df35d2c439757..8ecb591272492 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/export_rules.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/export_rules.ts @@ -411,6 +411,7 @@ function expectToMatchRuleSchema(obj: RuleResponse): void { severity: expect.any(String), output_index: expect.any(String), author: expect.arrayContaining([]), + license: expect.any(String), false_positives: expect.arrayContaining([]), from: expect.any(String), max_signals: expect.any(Number), diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/get_rule_params/get_custom_query_rule_params.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/get_rule_params/get_custom_query_rule_params.ts index a5c5fe00ed700..a27f99b6f75e8 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/get_rule_params/get_custom_query_rule_params.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/get_rule_params/get_custom_query_rule_params.ts @@ -30,6 +30,7 @@ export function getCustomQueryRuleParams( interval: '100m', from: 'now-6m', author: [], + license: 'Elastic License v2', enabled: false, ...rewrites, }; diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/create_prebuilt_rule_saved_objects.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/create_prebuilt_rule_saved_objects.ts index 20a8e6cf17280..3ebd928123cc4 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/create_prebuilt_rule_saved_objects.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/create_prebuilt_rule_saved_objects.ts @@ -9,11 +9,21 @@ import { Client } from '@elastic/elasticsearch'; import { PrebuiltRuleAsset } from '@kbn/security-solution-plugin/server/lib/detection_engine/prebuilt_rules'; import { getPrebuiltRuleMock, + getPrebuiltRuleMockOfType, getPrebuiltRuleWithExceptionsMock, } from '@kbn/security-solution-plugin/server/lib/detection_engine/prebuilt_rules/mocks'; +import type { TypeSpecificCreateProps } from '@kbn/security-solution-plugin/common/api/detection_engine'; import { ELASTIC_SECURITY_RULE_ID } from '@kbn/security-solution-plugin/common'; import { SECURITY_SOLUTION_SAVED_OBJECT_INDEX } from '@kbn/core-saved-objects-server'; +const ruleAssetSavedObjectESFields = { + type: 'security-rule', + references: [], + coreMigrationVersion: '8.6.0', + updated_at: '2022-11-01T12:56:39.717Z', + created_at: '2022-11-01T12:56:39.717Z', +}; + /** * A helper function to create a rule asset saved object * @@ -22,11 +32,20 @@ import { SECURITY_SOLUTION_SAVED_OBJECT_INDEX } from '@kbn/core-saved-objects-se */ export const createRuleAssetSavedObject = (overrideParams: Partial) => ({ 'security-rule': getPrebuiltRuleMock(overrideParams), - type: 'security-rule', - references: [], - coreMigrationVersion: '8.6.0', - updated_at: '2022-11-01T12:56:39.717Z', - created_at: '2022-11-01T12:56:39.717Z', + ...ruleAssetSavedObjectESFields, +}); + +/** + * A helper function to create a rule asset saved object + * + * @param overrideParams Params to override the default mock + * @returns Created rule asset saved object + */ +export const createRuleAssetSavedObjectOfType = ( + type: T['type'] +) => ({ + 'security-rule': getPrebuiltRuleMockOfType(type), + ...ruleAssetSavedObjectESFields, }); export const SAMPLE_PREBUILT_RULES = [ diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/index.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/index.ts index fbf9ab7b36384..fabd3df2f2d16 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/index.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/index.ts @@ -19,4 +19,4 @@ export * from './install_prebuilt_rules_fleet_package'; export * from './install_prebuilt_rules'; export * from './review_install_prebuilt_rules'; export * from './review_upgrade_prebuilt_rules'; -export * from './upgrade_prebuilt_rules'; +export * from './perform_upgrade_prebuilt_rules'; diff --git a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/upgrade_prebuilt_rules.ts b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/perform_upgrade_prebuilt_rules.ts similarity index 67% rename from x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/upgrade_prebuilt_rules.ts rename to x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/perform_upgrade_prebuilt_rules.ts index f12d0adbc65f3..c9b2543d61d69 100644 --- a/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/upgrade_prebuilt_rules.ts +++ b/x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/perform_upgrade_prebuilt_rules.ts @@ -7,8 +7,8 @@ import { PERFORM_RULE_UPGRADE_URL, - RuleVersionSpecifier, PerformRuleUpgradeResponseBody, + PerformRuleUpgradeRequestBody, } from '@kbn/security-solution-plugin/common/api/detection_engine/prebuilt_rules'; import type { Client } from '@elastic/elasticsearch'; import type SuperTest from 'supertest'; @@ -17,30 +17,21 @@ import { refreshSavedObjectIndices } from '../../refresh_index'; /** * Upgrades available prebuilt rules in Kibana. * - * - Pass in an array of rule version specifiers to upgrade specific rules. Otherwise - * all available rules will be upgraded. - * * @param supertest SuperTest instance - * @param rules Array of rule version specifiers to upgrade (optional) + * @param pazload Array of rule version specifiers to upgrade (optional) * @returns Upgrade prebuilt rules response */ -export const upgradePrebuiltRules = async ( +export const performUpgradePrebuiltRules = async ( es: Client, supertest: SuperTest.Agent, - rules?: RuleVersionSpecifier[] + requestBody: PerformRuleUpgradeRequestBody ): Promise => { - let payload = {}; - if (rules) { - payload = { mode: 'SPECIFIC_RULES', rules, pick_version: 'TARGET' }; - } else { - payload = { mode: 'ALL_RULES', pick_version: 'TARGET' }; - } const response = await supertest .post(PERFORM_RULE_UPGRADE_URL) .set('kbn-xsrf', 'true') .set('elastic-api-version', '1') .set('x-elastic-internal-origin', 'foo') - .send(payload) + .send(requestBody) .expect(200); await refreshSavedObjectIndices(es);