[SIEM] Removing a Rule's Action breaks Rule execution

[tsg]

[reported originally by @rylnd)

Kibana version: 7.7.0

After an action is removed from a rule, the rule fails to execute. To resolve this, disable and then re-enable the rule.

This is present on master and 7.7.

Steps to Reproduce

1. Create and activate a Rule with an Action
   • any action type and frequency work, but easiest to test with on each rule execution
   • similar to above, running the rule every 30s eases testing
2. (optional): wait for the rule to execute, generate signals, and fire the action
   • this is the pre-bug sanity check
3. Edit the Rule and "deactivate" its Action by changing its Actions Frequency to perform no actions
4. Observe errant behaviors:
   • Rule no longer generates signals
   • Rule no longer reports success/failure (Last response will remain as it was prior to step 3)
   • if running locally, kibana logs should show an error:

     server    log   [12:20:57.406] [error][encryptedSavedObjects][plugins] Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data
     server    log   [12:20:57.406] [error][alerting][alerting][plugins][plugins] Executing Alert "684c28a5-a49d-490b-9c9b-26dc865e641c" has resulted in Error: Unable to decrypt attribute "apiKey"
     

NB That querying the corresponding task, we find that it still exists in the "idle" state:

GET .kibana_task_manager*/_search
{"query":{"match":{"task.taskType":"alerting:siem.signals"}}}

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[rylnd]

This is a consequence of #53868, which is currently blocked and under discussion.

[rylnd]

Dev update: we have a temporary solution in the following diff:

diff --git a/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts b/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
index f15154a096..29dd52cc57 100644
--- a/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
+++ b/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
@@ -138,6 +138,7 @@ export const updateRulesRoute = (router: IRouter, ml: SetupPlugins['ml']) => {
             ruleActions,
             ruleStatuses.saved_objects[0]
           );
+          await alertsClient.updateApiKey({ id: rule.id });
           if (errors != null) {
             return siemResponse.error({ statusCode: 500, body: errors });
           } else {

It's not ideal as the call to updateApiKey itself logs an error when removing the rule's action, but it does allow the rule to continue executing sans actions. If acceptable we'll have to add a similar fix to all patch/update rule routes that can delete actions.

However, we're still digging into the underlying cause here, so we'll be roping in Alerting for some assistance before this ships.

[mikecote]

Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data usually means data has change since it was last encrypted. This can happen when doing a partial update or when the Elasticsearch document merges json on update. You can tell the latter by comparing alert params on update vs what is stored in the document in ES after update.

I don't think #53868 will solve the issue because that is to handle invalidating API keys where this issue is trying to decrypt the API key from a saved object.

[mikecote]

If it's confirmed that the alert params get merged with old values on update, this file x-pack/plugins/siem/server/lib/detection_engine/signals/signal_params_schema.ts would need to set default null values to all the optional properties.

[rylnd]

7.7: #67426

[MadameSheema]

Fixed in 7.7.1 :)