[Security Solution]An error "Rule failure" is getting displayed on Rule detail page.

[ghost]

Description:
An error "Rule failure" is getting displayed on Rule detail page.

Build Details:
Version: 7.14.0 BC3
Build: 42545
Commit: c314921
Artifact link: https://staging.elastic.co/7.14.0-682a8012/summary-7.14.0.html

Browser Details:
All

Preconditions:

• Kibana Environment should exist.
• Endpoint security and Elastic Agent should be installed
• Auditbeat should be running on 7.8.0 build
• Alert should generated

Steps to Reproduce:

1. Click on the Default space next to the navigation bar on 7.8.0 kibana version.
2. Click on "Manage Spaces"
3. Click on 'Create new space' button .
4. Add the Name and click on create new space button
5. Now, Create the custom rules with query "Host.name: *" to generate the signals with signal.rule.risk_score
6. Once the rule successfully generates the alerts, Navigate to Dev tools and run the following command.
   GET .<_index_name>/_mapping/field/signal.rule.risk_score i.e
7. Now upgrade the build from 7.8.0 to the current build 7.14.0
8. Now Navigate to Rules tab under Detect and click on 'Rule name'
9. Observe that an error "Rule failure" is getting displayed on Rule detail page.

Screen Shot

Actual Result:
An error "Rule failure" is getting displayed on Rule detail page.
Reason: unknown Message: params invalid: Invalid value "undefined" supplied to "author"

Expected Result:
No error should be displayed on clicking at Rule name

What's not working:

• NA

What's working:

• NA

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ghost]

@manishgupta-qasource Please review!

[manishgupta-qasource]

Reviewed & Assigned to @MadameSheema

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[MadameSheema]

@spong @peluja1012 can you please take a look at this? thanks :)

[spong]

So initial thought was that this was related to #101582 and a migration error on upgrade was resulting in the observed behavior. Upon looking at the kibana cluster logs, it looks like all went well with the migration and no errors were logged migrating either the kibana or task_manager indices.

Only other errors logged were when Kibana was coming back online and related to the cloud-internal-enterprise_search-server trying to hit kibana -- these disappeared once Kibana was available.

Only other notable log was with regard to Task Manager performance:

Detected potential performance issue with Task Manager. Set 'xpack.task_manager.monitored_stats_health_verbose_log.enabled: true' in your Kibana.yml to enable debug logging

Once Kibana came back online, the rule started failing immediately as described above:

08:42:37.000 Kibana is now available (was unavailable)
08:44:01.000 Executing Alert "bb9468fa-68cc-4d04-a7b7-3075506a8ae6" has resulted in Error: params invalid: Invalid value "undefined" supplied to "author"
08:44:07.000 Executing Alert "bb9468fa-68cc-4d04-a7b7-3075506a8ae6" has resulted in Error: params invalid: Invalid value "undefined" supplied to "author"
08:44:13.000 Executing Alert "bb9468fa-68cc-4d04-a7b7-3075506a8ae6" has resulted in Error: params invalid: Invalid value "undefined" supplied to "author"
08:44:19.000 Executing Alert "bb9468fa-68cc-4d04-a7b7-3075506a8ae6" has resulted in Error: params invalid: Invalid value "undefined" supplied to "author"

The frequency here is what stood out to me, and looking at the Rule configuration we can see it was set to a 5 second interval with a 1 minute lookback. This leads me to believe there was a good chance the rule was executing when the cluster was upgraded...so looking at those logs:

Last successful rule execution (as per Detection Engine Kibana logs) was at 08:31:14.000, then ES bounces for the upgrade:

Looking at the event-log for this period of time shows subsequent executions while ES is down:

---

So pretty safe to say this rule kept executing (or at least logging that it was) while ES was upgrading. @elastic/kibana-alerting-services, does this sound right to you? And if so, is there any chance this could result in similar issues to #101582 where the rule wouldn't be successfully migrated (although in this instance, without error)? Additionally, what is the expectation around enabled Rules and upgrades? Users shouldn't have to disable Rules prior to upgrading should they?

In the meantime @mandeepkaur-qasource, could you please try to reproduce but this time disabling the rule prior to upgrading? Thanks! 🙂

[ghost]

Hi @spong

We have validated this issue by de-activating the created custom rule before upgrading the 7.8.0 build to 7.14.0 and found that the below mentioned observation:
Case1: "Rule failure" error does not occur in case, if the rule is already De-Activated before upgrade.
Screen-Shots:

Case 2 But, If user tries to Activate this rule after upgrading, than an error "Your visualization has error(s)" is getting displayed.
Screen-Shots:

Steps followed

1. Click on the Default space next to the navigation bar on 7.8.0 kibana version.
2. Click on "Manage Spaces"
3. Click on 'Create new space' button .
4. Add the Name and click on create new space button
5. Now, Create the custom rules with query "Host.name: *" to generate the signals with signal.rule.risk_score
6. Once the rule successfully generates the alerts, Navigate to Dev tools and run the following command.
   GET .<_index_name>/_mapping/field/signal.rule.risk_score i.e
7. Now, De-activate the custom rule from Rule details page.
8. Upgrade the build from 7.8.0 to the current build 7.14.0
9. Now Navigate to Rules tab under Detect and click on Rule name
10. Now, click on toggle button to 'Activate' the rule.
11. Observe that an error "Your visualization has error(s)" is getting displayed.

Build Details:
Version: 7.14.0 BC4
Build: 42656
Commit: 82a4f6a
Artifact Page : https://staging.elastic.co/7.14.0-b3779639/summary-7.14.0.html

Thanks!!

[ymao1]

@spong Rules are not required to be disabled before upgrading.

We have discovered while working on #101582 that any error that occurs during the rule/connector migration process (decrypting, migrating, encrypting) will be caught, logged, and the unmigrated document returned. However, since there are no errors being logged during upgrade, it doesn't seem like there are errors occuring during the migration.

@mandeepkaur-qasource Can we get the rule document as it looked in 7.8 (either enabled or disabled) and how it looks in 7.14.0 after migration? Since the original document is from 7.8, all of our rule migrations (we have ones for 7.10, 7.11, 7.11.2 and 7.13) would be applied to the document in sequence so maybe something unexpected is happening during it.

The error message looks like it's coming from https://github.com/elastic/kibana/blob/master/x-pack/plugins/alerting/server/lib/validate_alert_type_params.ts which is called when a rule is created, updated or executed. It doesn't look like anything has changed in this code or the code that uses it (within the alerting framework) in several months. It is validating the params inside the rule document against the schema supplied by the rule type. Has anything changed about how the params schema is validated?

[ghost]

Hi @ymao1,
Just want to make sure that we are on the right track.
Do you want us to share the exported rule json files with both activated and decativated status on 7.8.0 and Than upgrade to 7.14.0 and again export the same rule and share that json as well?

[ymao1]

@mandeepkaur-qasource Can we get the documents from the .kibana index (vs exporting via the detection rule ui)? That would allow us to see the full saved object

[peluja1012]

@mandeepkaur-qasource you can get the documents from the .kibana index using Kibana Dev Tools as follows.

1. Export the rule using the UI
2. Grab the value for id in the exported ndjson file
3. Go to Dev tools and run the following command:

    GET .kibana*/_search
    {
      "query": {
        "match": {
          "_id": "alert:<id value from step 2>"
        }
      }
    }
   

[ghost]

Hi @peluja1012,

As requested, Please find all the related ndjsons files of exported Rules and documents that we got from the .kibana index using above mentioned steps from Kibana Dev Tools:

Please Find the observations listed below

Dev Tool: [7.8.0]

``` GET .kibana*/_search { "query": { "match": { "_id": "alert:676c2dc4-ee78-4626-b074-a88536430877" } } } ```

Dev Tool Result on 7.8.0:

Dev tool Result: [7.8.0]

``` { "took" : 334, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 0, "relation" : "eq" }, "max_score" : null, "hits" : [ ] } } ```

Screen-Shots:
Before Upgrade:

After upgrading the build from 7.8.0 to 7.14.0 please find the details fetched from Dev Tools console:

Dev Tool command: [7.14.0]

``` GET .kibana*/_search { "query": { "match": { "_id": "alert:676c2dc4-ee78-4626-b074-a88536430877" } } } ```

Dev Tool Result on 7.14.0:

Dev tool Results on 7.14.0

``` #! this request accesses system indices: [.kibana_1, .kibana_7.14.0_001, .kibana_security_session_1, .kibana_task_manager_1, .kibana_task_manager_7.14.0_001], but in a future major version, direct access to system indices will be prevented by default { "took" : 308, "timed_out" : false, "_shards" : { "total" : 7, "successful" : 7, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 0, "relation" : "eq" }, "max_score" : null, "hits" : [ ] } } ```

Screen-Shots:
After Upgrade:

7.8.0 Rule ndJson file:
rules_export.zip

7.14.0 Rule ndJson file:
rules_export (1).zip

Thanks!!

[ymao1]

@mandeepkaur-qasource It looks like those rules are in a space named new? If that is the case, can you try this:

GET .kibana*/_search { "query": { "match": { "_id": "new:alert:676c2dc4-ee78-4626-b074-a88536430877" } } }

Rules are stored in the .kibana index with id: ${spaceId}:alert:${ruleId}

[ghost]

Hi @ymao1

Yes, the rule was created with in a Space named as 'New'.
Please find all the related jsons files of Rule exported from the space before and after upgrade and documents that we got from the .kibana index with id: ${spaceId}:alert:${ruleId}

Please Find the observations listed below
7.8.0 Rule ndJson file:
rules_export (4).zip

7.14.0 Rule ndJson file:
rules_export (5).zip

7.8.0 Dev Tool Results:
7.8.0 Dev tool result.txt

7.8.0 Dev Tool Results:

{
  "took" : 313,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana_1",
        "_type" : "_doc",
        "_id" : "new:alert:f3f3c2d5-4cd0-42bc-8720-a6b2cc797b25",
        "_score" : 1.0,
        "_source" : {
          "alert" : {
            "name" : "Space_Rule",
            "tags" : [
              "__internal_rule_id:de798d45-3755-4532-bd6c-c82d1a4003af",
              "__internal_immutable:false"
            ],
            "alertTypeId" : "siem.signals",
            "consumer" : "siem",
            "params" : {
              "description" : "Test",
              "ruleId" : "de798d45-3755-4532-bd6c-c82d1a4003af",
              "index" : [
                "apm-*-transaction*",
                "auditbeat-*",
                "endgame-*",
                "filebeat-*",
                "packetbeat-*",
                "winlogbeat-*"
              ],
              "falsePositives" : [ ],
              "from" : "now-65s",
              "immutable" : false,
              "query" : "host.name : * ",
              "language" : "kuery",
              "outputIndex" : ".siem-signals-new",
              "savedId" : null,
              "timelineId" : null,
              "timelineTitle" : null,
              "meta" : {
                "from" : "1m",
                "kibana_siem_app_url" : "https://aacbde9659b84b68a92b8df09b530957.europe-west1.gcp.cloud.es.io:9243/s/new/app/siem"
              },
              "filters" : [ ],
              "maxSignals" : 100,
              "riskScore" : 50,
              "severity" : "low",
              "threat" : [ ],
              "to" : "now",
              "type" : "query",
              "references" : [ ],
              "note" : null,
              "version" : 1
            },
            "schedule" : {
              "interval" : "5s"
            },
            "enabled" : true,
            "actions" : [ ],
            "throttle" : null,
            "apiKeyOwner" : "elastic",
            "apiKey" : "xsEIg280UqS7oxSlpaBAi1aY22dASI80ES5hKAU1xaMfNCsEvzs8E82XYyJqrwhoJP50HGzJbqMI5zRKSlArOowD0KXrMTunHizUTjvnMRiB4J/aV0edt+RWbDwtUCuVJFHaM1e2l6wZzt6vBbK9MH+zGv6dDiL3EY5cl9wVmiR0eVl5SiW/J+Ue60EEmiDDnTo+VuYqhgVxMg==",
            "createdBy" : "elastic",
            "updatedBy" : "elastic",
            "createdAt" : "2021-07-28T04:57:51.164Z",
            "muteAll" : false,
            "mutedInstanceIds" : [ ],
            "scheduledTaskId" : "e0474c80-ef60-11eb-94c9-670b3b3d88ea"
          },
          "type" : "alert",
          "references" : [ ],
          "namespace" : "new",
          "updated_at" : "2021-07-28T05:01:32.386Z"
        }
      }
    ]
  }
}

7.14.0 Dev Tool Results:
7.14.0 Dev Tool Result.txt

7.14.0 Dev Tool Results

#! this request accesses system indices: [.kibana_1, .kibana_7.14.0_001, .kibana_security_session_1, .kibana_task_manager_1, .kibana_task_manager_7.14.0_001], but in a future major version, direct access to system indices will be prevented by default
{
  "took" : 666,
  "timed_out" : false,
  "_shards" : {
    "total" : 7,
    "successful" : 7,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana_1",
        "_type" : "_doc",
        "_id" : "new:alert:f3f3c2d5-4cd0-42bc-8720-a6b2cc797b25",
        "_score" : 1.0,
        "_source" : {
          "alert" : {
            "name" : "Space_Rule",
            "tags" : [
              "__internal_rule_id:de798d45-3755-4532-bd6c-c82d1a4003af",
              "__internal_immutable:false"
            ],
            "alertTypeId" : "siem.signals",
            "consumer" : "siem",
            "params" : {
              "description" : "Test",
              "ruleId" : "de798d45-3755-4532-bd6c-c82d1a4003af",
              "index" : [
                "apm-*-transaction*",
                "auditbeat-*",
                "endgame-*",
                "filebeat-*",
                "packetbeat-*",
                "winlogbeat-*"
              ],
              "falsePositives" : [ ],
              "from" : "now-65s",
              "immutable" : false,
              "query" : "host.name : * ",
              "language" : "kuery",
              "outputIndex" : ".siem-signals-new",
              "savedId" : null,
              "timelineId" : null,
              "timelineTitle" : null,
              "meta" : {
                "from" : "1m",
                "kibana_siem_app_url" : "https://aacbde9659b84b68a92b8df09b530957.europe-west1.gcp.cloud.es.io:9243/s/new/app/siem"
              },
              "filters" : [ ],
              "maxSignals" : 100,
              "riskScore" : 50,
              "severity" : "low",
              "threat" : [ ],
              "to" : "now",
              "type" : "query",
              "references" : [ ],
              "note" : null,
              "version" : 1
            },
            "schedule" : {
              "interval" : "5s"
            },
            "enabled" : true,
            "actions" : [ ],
            "throttle" : null,
            "apiKeyOwner" : "elastic",
            "apiKey" : "xsEIg280UqS7oxSlpaBAi1aY22dASI80ES5hKAU1xaMfNCsEvzs8E82XYyJqrwhoJP50HGzJbqMI5zRKSlArOowD0KXrMTunHizUTjvnMRiB4J/aV0edt+RWbDwtUCuVJFHaM1e2l6wZzt6vBbK9MH+zGv6dDiL3EY5cl9wVmiR0eVl5SiW/J+Ue60EEmiDDnTo+VuYqhgVxMg==",
            "createdBy" : "elastic",
            "updatedBy" : "elastic",
            "createdAt" : "2021-07-28T04:57:51.164Z",
            "muteAll" : false,
            "mutedInstanceIds" : [ ],
            "scheduledTaskId" : "e0474c80-ef60-11eb-94c9-670b3b3d88ea"
          },
          "type" : "alert",
          "references" : [ ],
          "namespace" : "new",
          "updated_at" : "2021-07-28T05:01:32.386Z"
        }
      },
      {
        "_index" : ".kibana_7.14.0_001",
        "_type" : "_doc",
        "_id" : "new:alert:f3f3c2d5-4cd0-42bc-8720-a6b2cc797b25",
        "_score" : 1.0,
        "_source" : {
          "alert" : {
            "name" : "Space_Rule",
            "tags" : [
              "__internal_rule_id:de798d45-3755-4532-bd6c-c82d1a4003af",
              "__internal_immutable:false"
            ],
            "alertTypeId" : "siem.signals",
            "consumer" : "siem",
            "params" : {
              "description" : "Test",
              "ruleId" : "de798d45-3755-4532-bd6c-c82d1a4003af",
              "index" : [
                "apm-*-transaction*",
                "auditbeat-*",
                "endgame-*",
                "filebeat-*",
                "packetbeat-*",
                "winlogbeat-*"
              ],
              "falsePositives" : [ ],
              "from" : "now-65s",
              "immutable" : false,
              "query" : "host.name : * ",
              "language" : "kuery",
              "outputIndex" : ".siem-signals-new",
              "meta" : {
                "from" : "1m",
                "kibana_siem_app_url" : "https://aacbde9659b84b68a92b8df09b530957.europe-west1.gcp.cloud.es.io:9243/s/new/app/siem"
              },
              "filters" : [ ],
              "maxSignals" : 100,
              "riskScore" : 50,
              "severity" : "low",
              "threat" : [ ],
              "to" : "now",
              "type" : "query",
              "references" : [ ],
              "version" : 1,
              "riskScoreMapping" : [ ],
              "severityMapping" : [ ],
              "exceptionsList" : [ ]
            },
            "schedule" : {
              "interval" : "5s"
            },
            "enabled" : true,
            "actions" : [ ],
            "throttle" : null,
            "apiKeyOwner" : "elastic",
            "apiKey" : "4g+xQzBhI6iKei5ag0g7wd1OVbS3ADbV3NA237OvUxoGY3hcXGXSj/uSpW+t6q1CZW8OYgAc1US79ftSWLILcfUCQWTTTQXwvieD6ucUfoL12qv4J02JGHTi3isKl/kcv8oWR7xM3df5C+6BY65Xaywi4kwWB4R17kk3eEtvATXIkZgbnuxApDHyNMMGWjd1sotKflC1XOOPEQ==",
            "createdBy" : "elastic",
            "updatedBy" : "elastic",
            "createdAt" : "2021-07-28T04:57:51.164Z",
            "muteAll" : false,
            "mutedInstanceIds" : [ ],
            "scheduledTaskId" : "e0474c80-ef60-11eb-94c9-670b3b3d88ea",
            "meta" : {
              "versionApiKeyLastmodified" : "pre-7.10.0"
            },
            "executionStatus" : {
              "status" : "error",
              "lastExecutionDate" : "2021-07-28T06:09:44.530Z",
              "error" : {
                "reason" : "unknown",
                "message" : "params invalid: Invalid value \"undefined\" supplied to \"author\""
              }
            },
            "updatedAt" : "2021-07-28T05:01:32.386Z",
            "notifyWhen" : "onActiveAlert"
          },
          "type" : "alert",
          "references" : [ ],
          "namespace" : "new",
          "migrationVersion" : {
            "alert" : "7.13.0"
          },
          "coreMigrationVersion" : "7.14.0",
          "updated_at" : "2021-07-28T06:09:44.580Z"
        }
      }
    ]
  }
}

Thanks!!

[ymao1]

@mandeepkaur-qasource @spong

Thanks for supplying the alerting saved object! I have loaded it into our framework tests and confirmed that all alerting migrations (we have ones for 7.10, 7.11 and 7.13) run successfully with no errors. In addition, looking at the supplied 7.14 alerting saved object, I can confirm that migrations ran on this saved object as well.

It seems like rules from 7.8 do not fill in the params.author field. Since none of the migrations add the params.author field, the detection rule is migrated to 7.14 without the params.author field. Looking at the definition for ruleParams in x-pack/plugins/security_solution/server/lib/detection_engine/schemas/rule_schemas.ts in 7.14, author is a required field (expects an array of strings), which leads to the error you're running into. At the start of each rule execution, the params schema validation is applied and since ruleParams specifies that author cannot be undefined, this validation error is thrown.

I see there is a security rule migration for 7.13. Perhaps there needed to be something in that migration that injected an empty array for author if author was undefined?

[MadameSheema]

@mandeepkaur-qasource is this happening just from 7.8 to 7.14? 7.10 to 7.14 works fine? thanks :)

[peluja1012]

Thanks, @ymao1 for performing such a thorough investigation!

I was not able to reproduce this issue when upgrading from 7.9 -> 7.13, from 7.10 -> 7.13, or from 7.12 ->7.13. I was only able to reproduce when upgrading from 7.8 -> 7.13. When upgrading from 7.18 -> 7.13, the reproduction steps are even simpler as you can see below:

1. In a 7.8 environment, navigate to the Security Solution -> Detections page and create a custom query rule.
2. Deactivate the rule
3. Upgrade from 7.8 to 7.13
4. Navigate to the rule details page for this rule and Activate it.
5. Observe that an error toaster message gets displayed with this content: Your visualization has error(s)" is getting displayed....

I was also able to reproduce the rule execution error in the description of this issue by following these steps:

1. In a 7.8 environment, navigate to the Security Solution -> Detections page and create a custom query rule.
2. Activate the rule
3. Upgrade from 7.8 to 7.13
4. Navigate to the rule details page for this rule.
5. Observe that a rule execution error at the top of the rule details page with this content Reason: unknown Message: params invalid: Invalid value "undefined" supplied to "author"

The workaround is to use the PATCH rule API to update each rule that has this issue. The payload of the PATCH call should set the author field to [] as follows. After the author field is populated the rule will start working as expected.

PATCH <kibana host>:<port>/api/detection_engine/rules
{
  "id": <id-value-of-rule>",
  "author": []
}

To fix this issue, I think we will need to add a migration that sets the author field.

[peluja1012]

@mandeepkaur-qasource Could you please validate this in the first BC for 7.15? We merged this PR to fix it #107230

[MadameSheema]

@deepikakeshav-qasource @karanbirsingh-qasource can you please check if this is still happening on 7.15BC1? thanks

[ghost]

Hi @MadameSheema ,

We have validated this ticket on 7.15.0 BC1 build and observed that issue is Fixed. No error is displyaing when activate and deactivate the rule after upgrade the build from 7.8.0 to 7.15.0

Build Details:

Version : 7.15.0 BC1
Commit:d791226d9385122f33f4a5ca38fa5369012fbec3
Build: 43636

Screenshot:

7.8.0: Rule created

7.15.0:

Hence, we are closing this issue and marking as QA Validated.

Thanks!!