Detect rule param changes for rolling upgrades and rollback assessment #173936

mikecote · 2023-12-22T19:33:12Z

In this PR, I'm adding a test in the alerting framework to detect changes in a rule type's params schema that will require a snapshot to be updated. This snapshot will provide a centralized place to view history on alerting rule params in case we need to asses risk for rolling upgrades or rollbacks of a release (serverless). The only rule types affected are those running in serverless in any of the three project types.

When a rule type is used in serverless, it must provide one of the following configuration to their rule type on top of everything else:

// Zod schema
schemas: {
  params: {
    type: 'zod',
    schema: UnifiedQueryRuleParams
  },
},

// config-schema
schemas: {
  params: {
    type: 'config-schema',
    schema: EsQueryRuleParamsSchema,
  },
},

We are working on documenting guidelines so engineers and response ops can ensure a change to rule parameters will work properly in rolling upgrade and rollback scenarios and be part of the PR review process.

NOTE to rule type owners: I pass the same schema used to validate to the schemas.params attribute in the rule type. It will be important to keep them in sync. Down the road, we plan to make validate.params optional and use the schema as a starting point so it's easier to have a single variable passed in.

To verify

Make changes to the params schema of the ES query rule type.

diff --git a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
index 73e8eae32cf..09ec74104ec 100644
--- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
+++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
@@ -39,6 +39,7 @@ export type EsQueryRuleParamsExtractedParams = Omit<EsQueryRuleParams, 'searchCo
 };

 const EsQueryRuleParamsSchemaProperties = {
+  foo: schema.boolean(),
   size: schema.number({ min: 0, max: ES_QUERY_MAX_HITS_PER_EXECUTION }),
   timeWindowSize: schema.number({ min: 1 }),
   excludeHitsFromPreviousRun: schema.boolean({ defaultValue: true }),

Run the jest integration test to update the snapshot file

node scripts/jest_integration.js x-pack/plugins/alerting/server/integration_tests/serverless_upgrade_and_rollback_checks.test.ts -u

Notice the x-pack/plugins/alerting/server/integration_tests/__snapshots__/serverless_upgrade_and_rollback_checks.test.ts.snap file got updated

    "foo": Object {
      "flags": Object {
        "error": [Function],
      },
      "type": "boolean",
    },

…-param-changes

mikecote · 2024-01-05T15:48:32Z

/ci

…-fix'

…ibana into poc/detect-rule-param-changes

mikecote · 2024-01-05T17:38:22Z

@elasticmachine merge upstream

mikecote · 2024-01-05T17:38:40Z

/ci

…fix'

mikecote · 2024-01-05T17:59:35Z

/ci

…ibana into poc/detect-rule-param-changes

mikecote · 2024-01-09T17:57:57Z

/ci

…-param-changes

mikecote · 2024-01-15T16:16:07Z

/ci

elasticmachine · 2024-01-17T00:09:42Z

Pinging @elastic/response-ops (Team:ResponseOps)

mikecote · 2024-01-17T11:25:54Z

@elasticmachine merge upstream

ymao1

Response ops changes LGTM

elasticmachine · 2024-01-17T14:26:52Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

elasticmachine · 2024-01-17T14:26:52Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

nreese

kibana-gis changes LGTM
code review only

rylnd

I reviewed changes to Detection Engine code. Both the referenced schema and their associated types look correct, as well as the relevant snapshot tests. LGTM!

jennypavlova

Obs UX changes LGTM 💯 I checked the rules/alerts locally and tested it as described.

darnautov

ML and Transform changes LGTM

kibana-ci · 2024-02-05T21:08:23Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 0ab6ea4

Failed CI Steps

FTR Configs #70

Test Failures

[job] [logs] FTR Configs #70 / Reporting APIs CSV Generation from SearchSource "before all" hook in "CSV Generation from SearchSource"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`alerting`	803	804	+1

Unknown metric groups

API count

id	before	after	diff
`alerting`	834	835	+1

History

💛 Build #187874 was flaky f37ad20
💛 Build #187527 was flaky a80cfe1
💔 Build #186639 failed 6f06f82
💔 Build #186181 failed b7d7259
💔 Build #186176 failed 397d44a
💔 Build #186141 failed b3cf173

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

elastic#173936) In this PR, I'm adding a test in the alerting framework to detect changes in a rule type's params schema that will require a snapshot to be updated. This snapshot will provide a centralized place to view history on alerting rule params in case we need to asses risk for rolling upgrades or rollbacks of a release (serverless). The only rule types affected are those running in serverless in any of the three project types. When a rule type is used in serverless, it must provide one of the following configuration to their rule type on top of everything else: ``` // Zod schema schemas: { params: { type: 'zod', schema: UnifiedQueryRuleParams }, }, // config-schema schemas: { params: { type: 'config-schema', schema: EsQueryRuleParamsSchema, }, }, ``` We are working on documenting guidelines so engineers and response ops can ensure a change to rule parameters will work properly in rolling upgrade and rollback scenarios and be part of the PR review process. NOTE to rule type owners: I pass the same schema used to validate to the `schemas.params` attribute in the rule type. It will be important to keep them in sync. Down the road, we plan to make `validate.params` optional and use the schema as a starting point so it's easier to have a single variable passed in. ## To verify 1. Make changes to the params schema of the ES query rule type. ``` diff --git a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts index 73e8eae32cf..09ec74104ec 100644 --- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts +++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts @@ -39,6 +39,7 @@ export type EsQueryRuleParamsExtractedParams = Omit<EsQueryRuleParams, 'searchCo }; const EsQueryRuleParamsSchemaProperties = { + foo: schema.boolean(), size: schema.number({ min: 0, max: ES_QUERY_MAX_HITS_PER_EXECUTION }), timeWindowSize: schema.number({ min: 1 }), excludeHitsFromPreviousRun: schema.boolean({ defaultValue: true }), ``` 2. Run the jest integration test to update the snapshot file ``` node scripts/jest_integration.js x-pack/plugins/alerting/server/integration_tests/serverless_upgrade_and_rollback_checks.test.ts -u ``` 3. Notice the `x-pack/plugins/alerting/server/integration_tests/__snapshots__/serverless_upgrade_and_rollback_checks.test.ts.snap` file got updated ``` "foo": Object { "flags": Object { "error": [Function], }, "type": "boolean", }, ``` --------- Co-authored-by: kibanamachine <[email protected]>

mikecote added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Dec 22, 2023

mikecote force-pushed the poc/detect-rule-param-changes branch from 69d8613 to bc465a3 Compare December 27, 2023 15:21

Initial commit

7c3b37f

mikecote force-pushed the poc/detect-rule-param-changes branch from 97a94c9 to 7c3b37f Compare December 28, 2023 13:56

mikecote added 7 commits January 3, 2024 12:28

Merge branch 'main' of github.com:elastic/kibana into poc/detect-rule…

dee2174

…-param-changes

Merge branch 'main' of github.com:elastic/kibana into poc/detect-rule…

b21a7de

…-param-changes

Merge branch 'main' of github.com:elastic/kibana into poc/detect-rule…

f4b2b88

…-param-changes

Narrow checks for serverless rule types

ca9fbc9

Add section on rule type definition to define a schema

25cfd04

Cleanup

d2a525d

Edit comment

b3cf173

kibanamachine and others added 3 commits January 5, 2024 16:18

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

9babc18

…-fix'

Fix package.json

41cbb73

Merge branch 'poc/detect-rule-param-changes' of github.com:mikecote/k…

d350fad

…ibana into poc/detect-rule-param-changes

Merge branch 'main' into poc/detect-rule-param-changes

397d44a

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

b7d7259

…fix'

mikecote added 4 commits January 5, 2024 14:13

Update yarn lock changes

264c1bd

Merge branch 'poc/detect-rule-param-changes' of github.com:mikecote/k…

63806ff

…ibana into poc/detect-rule-param-changes

Add a ts ignore

6bb539f

Update with main

6f06f82

mikecote added 2 commits January 15, 2024 11:10

Merge branch 'main' of github.com:elastic/kibana into poc/detect-rule…

4055938

…-param-changes

Fix typecheck errors

a80cfe1

Remove TS ignore

2361633

mikecote requested a review from a team as a code owner January 17, 2024 00:09

mikecote requested a review from rylnd January 17, 2024 00:09

Merge branch 'main' into poc/detect-rule-param-changes

f37ad20

mikecote mentioned this pull request Jan 17, 2024

Notify the response ops team whenever a change is made to connector config, secrets or params schema #175018

Closed

ymao1 approved these changes Jan 17, 2024

View reviewed changes

botelastic bot added Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-management Observability Management User Experience Team labels Jan 17, 2024

nreese approved these changes Jan 17, 2024

View reviewed changes

smith added the apm:review label Jan 17, 2024

rylnd approved these changes Jan 18, 2024

View reviewed changes

jennypavlova self-requested a review January 22, 2024 09:37

jennypavlova approved these changes Jan 22, 2024

View reviewed changes

darnautov approved these changes Jan 24, 2024

View reviewed changes

kpatticha approved these changes Jan 29, 2024

View reviewed changes

simianhacker self-requested a review February 1, 2024 16:43

simianhacker approved these changes Feb 1, 2024

View reviewed changes

mikecote added 2 commits February 5, 2024 14:45

Update with main

70cd9ef

Update snapshot

0ab6ea4

mikecote enabled auto-merge (squash) February 5, 2024 19:53

mikecote merged commit 9f1f142 into elastic:main Feb 5, 2024
39 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect rule param changes for rolling upgrades and rollback assessment #173936

Detect rule param changes for rolling upgrades and rollback assessment #173936

mikecote commented Dec 22, 2023 •

edited

Loading

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 9, 2024

mikecote commented Jan 15, 2024

elasticmachine commented Jan 17, 2024

mikecote commented Jan 17, 2024

ymao1 left a comment

elasticmachine commented Jan 17, 2024

elasticmachine commented Jan 17, 2024

nreese left a comment

rylnd left a comment

jennypavlova left a comment

darnautov left a comment

kibana-ci commented Feb 5, 2024

API count

Detect rule param changes for rolling upgrades and rollback assessment #173936

Detect rule param changes for rolling upgrades and rollback assessment #173936

Conversation

mikecote commented Dec 22, 2023 • edited Loading

To verify

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 5, 2024

mikecote commented Jan 9, 2024

mikecote commented Jan 15, 2024

elasticmachine commented Jan 17, 2024

mikecote commented Jan 17, 2024

ymao1 left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 17, 2024

elasticmachine commented Jan 17, 2024

nreese left a comment

Choose a reason for hiding this comment

rylnd left a comment

Choose a reason for hiding this comment

jennypavlova left a comment

Choose a reason for hiding this comment

darnautov left a comment

Choose a reason for hiding this comment

kibana-ci commented Feb 5, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

mikecote commented Dec 22, 2023 •

edited

Loading