Skip to content

Commit

Permalink
[8.x] [Security Solution] Test plan for rule `type` field d…
Browse files Browse the repository at this point in the history 
…iff algorithm (#193372) (#194459)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Test plan for rule `type` field diff
algorithm (#193372)](#193372)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Davis
Plumlee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-30T16:37:29Z","message":"[Security
Solution] Test plan for rule `type` field diff algorithm (#193372)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/190482\r\n\r\nAdds test plan
for diff algorithm for `type` field diff algorithm\r\nimplemented here:
https://github.com/elastic/kibana/pull/193369\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fefa59f41206c534297813af2cb6f732c2c59aeb","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:prev-minor","v8.16.0"],"title":"[Security Solution]
Test plan for rule `type` field diff
algorithm","number":193372,"url":"https://github.com/elastic/kibana/pull/193372","mergeCommit":{"message":"[Security
Solution] Test plan for rule `type` field diff algorithm (#193372)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/190482\r\n\r\nAdds test plan
for diff algorithm for `type` field diff algorithm\r\nimplemented here:
https://github.com/elastic/kibana/pull/193369\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fefa59f41206c534297813af2cb6f732c2c59aeb"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193372","number":193372,"mergeCommit":{"message":"[Security
Solution] Test plan for rule `type` field diff algorithm (#193372)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/190482\r\n\r\nAdds test plan
for diff algorithm for `type` field diff algorithm\r\nimplemented here:
https://github.com/elastic/kibana/pull/193369\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fefa59f41206c534297813af2cb6f732c2c59aeb"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Davis Plumlee <[email protected]>
  • Loading branch information
@kibanamachine @dplumlee
kibanamachine and dplumlee authored Sep 30, 2024
1 parent 125f378 commit e6593a0
Showing 1 changed file with 109 additions and 8 deletions.
117 changes: 109 additions & 8 deletions ...sting/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,16 @@ Status: `in progress`.
- [Rule field doesn't have an update and has no custom value - `AAA`](#rule-field-doesnt-have-an-update-and-has-no-custom-value---aaa)
- [**Scenario: `AAA` - Rule field is any type**](#scenario-aaa---rule-field-is-any-type)
- [Rule field doesn't have an update but has a custom value - `ABA`](#rule-field-doesnt-have-an-update-but-has-a-custom-value---aba)
- [**Scenario: `ABA` - Rule field is any type**](#scenario-aba---rule-field-is-any-type)
- [**Scenario: `ABA` - Rule field is any type except rule `type`**](#scenario-aba---rule-field-is-any-type-except-rule-type)
- [**Scenario: `ABA` - Rule field is rule `type`**](#scenario-aba---rule-field-is-rule-type)
- [Rule field has an update and doesn't have a custom value - `AAB`](#rule-field-has-an-update-and-doesnt-have-a-custom-value---aab)
- [**Scenario: `AAB` - Rule field is any type**](#scenario-aab---rule-field-is-any-type)
- [**Scenario: `AAB` - Rule field is any type except rule `type`**](#scenario-aab---rule-field-is-any-type-except-rule-type)
- [**Scenario: `AAB` - Rule field is rule `type`**](#scenario-aab---rule-field-is-rule-type)
- [Rule field has an update and a custom value that are the same - `ABB`](#rule-field-has-an-update-and-a-custom-value-that-are-the-same---abb)
- [**Scenario: `ABB` - Rule field is any type**](#scenario-abb---rule-field-is-any-type)
- [**Scenario: `ABB` - Rule field is any type except rule `type`**](#scenario-abb---rule-field-is-any-type-except-rule-type)
- [**Scenario: `ABB` - Rule field is rule `type`**](#scenario-abb---rule-field-is-rule-type)
- [Rule field has an update and a custom value that are NOT the same - `ABC`](#rule-field-has-an-update-and-a-custom-value-that-are-not-the-same---abc)
- [**Scenario: `ABC` - Rule field is rule `type`**](#scenario-abc---rule-field-is-rule-type)
- [**Scenario: `ABC` - Rule field is a number or single line string**](#scenario-abc---rule-field-is-a-number-or-single-line-string)
- [**Scenario: `ABC` - Rule field is a mergeable multi line string**](#scenario-abc---rule-field-is-a-mergeable-multi-line-string)
- [**Scenario: `ABC` - Rule field is a non-mergeable multi line string**](#scenario-abc---rule-field-is-a-non-mergeable-multi-line-string)
Expand All @@ -37,6 +41,7 @@ Status: `in progress`.
- [**Scenario: `-AB` - Rule field is an array of scalar values**](#scenario--ab---rule-field-is-an-array-of-scalar-values)
- [**Scenario: `-AB` - Rule field is a solvable `data_source` object**](#scenario--ab---rule-field-is-a-solvable-data_source-object)
- [**Scenario: `-AB` - Rule field is a non-solvable `data_source` object**](#scenario--ab---rule-field-is-a-non-solvable-data_source-object)
- [**Scenario: `-AB` - Rule field is rule `type`**](#scenario--ab---rule-field-is-rule-type)

## Useful information

Expand Down Expand Up @@ -74,7 +79,7 @@ Status: `in progress`.

#### **Scenario: `AAA` - Rule field is any type**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm
**Automation**: 11 integration tests with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is not customized by the user (current version == base version)
Expand All @@ -85,6 +90,7 @@ And <field_name> field should not be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "query" | "query" | "query" |
| single line string | name | "A" | "A" | "A" | "A" |
| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." |
| number | risk_score | 1 | 1 | 1 | 1 |
Expand All @@ -99,7 +105,7 @@ Examples:

### Rule field doesn't have an update but has a custom value - `ABA`

#### **Scenario: `ABA` - Rule field is any type**
#### **Scenario: `ABA` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -124,9 +130,27 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `ABA` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "query" | "query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and doesn't have a custom value - `AAB`

#### **Scenario: `AAB` - Rule field is any type**
#### **Scenario: `AAB` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -151,9 +175,27 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `AAB` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is not customized by the user (current version == base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and a custom value that are the same - `ABB`

#### **Scenario: `ABB` - Rule field is any type**
#### **Scenario: `ABB` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -179,8 +221,46 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `ABB` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and a custom value that are NOT the same - `ABC`

#### **Scenario: `ABC` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithms

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "threshold" | "threshold" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered. This test case scenario cannot currently be reached.

#### **Scenario: `ABC` - Rule field is a number or single line string**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithms
Expand Down Expand Up @@ -328,7 +408,7 @@ Examples:

#### **Scenario: `-AA` - Rule field is any type**

**Automation**: 9 integration tests with mock rules + a set of unit tests for each algorithm
**Automation**: 11 integration tests with mock rules + a set of unit tests for each algorithm

```Gherkin
Given at least 1 installed prebuilt rule has a new version available
Expand All @@ -340,6 +420,7 @@ And <field_name> field should not be shown in the upgrade preview UI
Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | N/A | "query" | "query" | "query" |
| single line string | name | N/A | "A" | "A" | "A" |
| multi line string | description | N/A | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." |
| number | risk_score | N/A | 1 | 1 | 1 |
Expand Down Expand Up @@ -438,3 +519,23 @@ Examples:
| algorithm | base_version | current_version | target_version | merged_version |
| data_source | N/A | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} |
```

#### **Scenario: `-AB` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 installed prebuilt rule has a new version available
And the base version of the rule cannot be determined
And customized data_source field is different than the Elastic update in this upgrade (current version != target version)
And current version and target version are not both array fields in data_source
Then for data_source field the diff algorithm should output the target version as the merged version with a non-solvable conflict
And data_source field should be returned from the `upgrade/_review` API endpoint
And data_source field should be shown in the upgrade preview UI
Examples:
| algorithm | base_version | current_version | target_version | merged_version |
| rule type | N/A | "query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

0 comments on commit e6593a0

Please sign in to comment.