[ResponseOps][Actions] support mustache context variables with periods in the name

[doakalexi]

Resolves #127748

Summary

When we encounter context fields with a . in the name we expand it to an object and add it back to the context.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

To verify

• Create a rule and add a field with . to the action message mustache template. Verify that the message contains the field value when the rule runs.

This is what I did to verify, but you can choose to do what ever is easiest!

• I created a new index by doing the following:

PUT _ingest/pipeline/add-timestamp
{
  "processors" : [
    {
      "set" : {
        "field": "@timestamp",
        "value": "{{{_ingest.timestamp}}}"
      }
    }
  ]
}

POST my-index/_doc/?pipeline=add-timestamp
{
   "my.field": "test"
}

I kept adding documents to make sure my rule would hit on something.

• Create an elasticsearch rule with the following query:

with a sever log action that had the following message:

Elasticsearch query alert is active:
{{#context.hits}}
_source.my.field: {{_source.my.field}}
{{/context.hits}}

• Verify that when the rule is active the server log message is populated with the values in _source.my.field

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

When I follow your verification steps, I do see that this is working but when I tried to create an ES query rule agains an alerts as data index, I'm not seeing it work.

Here's what my alerts as data doc looks like:

"hits": [
      {
        "_index": ".internal.alerts-observability.metrics.alerts-default-000001",
        "_id": "a8616ced-c22b-466c-a964-8db53af930ef",
        "_score": 1,
        "_source": {
          "kibana.alert.rule.category": "Metric threshold",
          "kibana.alert.rule.consumer": "infrastructure",
          "kibana.alert.rule.execution.uuid": "c42da290-30be-4e90-a7fb-75e160bac758",
          "kibana.alert.rule.name": "test rule",
          "kibana.alert.rule.producer": "infrastructure",
          "kibana.alert.rule.rule_type_id": "metrics.alert.threshold",
          "kibana.alert.rule.uuid": "534c0f20-5533-11ed-b0da-c1155191eec9",
          "kibana.space_ids": [
            "default"
          ],
          "kibana.alert.rule.tags": [],
          "@timestamp": "2022-10-26T13:50:06.516Z",
          "kibana.alert.reason": "event.duration is 235,545,454.54545 in the last 1 min for execute. Alert when > 0.",
          "kibana.alert.duration.us": 759925000,
          "kibana.alert.time_range": {
            "gte": "2022-10-26T13:37:26.591Z"
          },
          "kibana.alert.instance.id": "execute",
          "kibana.alert.start": "2022-10-26T13:37:26.591Z",
          "kibana.alert.uuid": "a8616ced-c22b-466c-a964-8db53af930ef",
          "kibana.alert.status": "active",
          "kibana.alert.workflow_status": "open",
          "event.kind": "signal",
          "event.action": "active",
          "kibana.version": "8.6.0",
          "tags": []
        }
      },
      {
        "_index": ".internal.alerts-observability.metrics.alerts-default-000001",
        "_id": "6f7df590-f3b2-4c10-89f6-eb45b47f283c",
        "_score": 1,
        "_source": {
          "kibana.alert.rule.category": "Metric threshold",
          "kibana.alert.rule.consumer": "infrastructure",
          "kibana.alert.rule.execution.uuid": "c42da290-30be-4e90-a7fb-75e160bac758",
          "kibana.alert.rule.name": "test rule",
          "kibana.alert.rule.producer": "infrastructure",
          "kibana.alert.rule.rule_type_id": "metrics.alert.threshold",
          "kibana.alert.rule.uuid": "534c0f20-5533-11ed-b0da-c1155191eec9",
          "kibana.space_ids": [
            "default"
          ],
          "kibana.alert.rule.tags": [],
          "@timestamp": "2022-10-26T13:50:06.516Z",
          "kibana.alert.reason": "event.duration is 11,885,000,000 in the last 1 min for recovered-instance. Alert when > 0.",
          "kibana.alert.duration.us": 759925000,
          "kibana.alert.time_range": {
            "gte": "2022-10-26T13:37:26.591Z"
          },
          "kibana.alert.instance.id": "recovered-instance",
          "kibana.alert.start": "2022-10-26T13:37:26.591Z",
          "kibana.alert.uuid": "6f7df590-f3b2-4c10-89f6-eb45b47f283c",
          "kibana.alert.status": "active",
          "kibana.alert.workflow_status": "open",
          "event.kind": "signal",
          "event.action": "active",
          "kibana.version": "8.6.0",
          "tags": []
        }
      },
      {
        "_index": ".internal.alerts-observability.metrics.alerts-default-000001",
        "_id": "a4d5a266-b1fd-47e2-b43e-bf516dba3eb9",
        "_score": 1,
        "_source": {
          "kibana.alert.rule.category": "Metric threshold",
          "kibana.alert.rule.consumer": "infrastructure",
          "kibana.alert.rule.execution.uuid": "c42da290-30be-4e90-a7fb-75e160bac758",
          "kibana.alert.rule.name": "test rule",
          "kibana.alert.rule.producer": "infrastructure",
          "kibana.alert.rule.rule_type_id": "metrics.alert.threshold",
          "kibana.alert.rule.uuid": "534c0f20-5533-11ed-b0da-c1155191eec9",
          "kibana.space_ids": [
            "default"
          ],
          "kibana.alert.rule.tags": [],
          "@timestamp": "2022-10-26T13:50:06.516Z",
          "kibana.alert.reason": "event.duration is 21,835,964,285.71429 in the last 1 min for active-instance. Alert when > 0.",
          "kibana.alert.duration.us": 251380000,
          "kibana.alert.time_range": {
            "gte": "2022-10-26T13:45:55.136Z"
          },
          "kibana.alert.instance.id": "active-instance",
          "kibana.alert.start": "2022-10-26T13:45:55.136Z",
          "kibana.alert.uuid": "a4d5a266-b1fd-47e2-b43e-bf516dba3eb9",
          "kibana.alert.status": "active",
          "kibana.alert.workflow_status": "open",
          "event.kind": "signal",
          "event.action": "active",
          "kibana.version": "8.6.0",
          "tags": []
        }
      }
    ]

And I have this in my action:

{{#context.hits}}
  category: {{_source.kibana.alert.rule.category}}
{{/context.hits}}

I see this in my sever log:

[2022-10-26T09:54:12.681-04:00][INFO ][plugins.actions.server-log] Server log: Elasticsearch query alert 'test dots' is active:;;- Value: 3;- Conditions Met: Number of matching documents is greater than 0 over 8d;- Timestamp: 2022-10-26T13:54:09.688Z;- Link: /app/management/insightsAndAlerting/triggersActions/rule/4ad3d390-5534-11ed-b0da-c1155191eec9;;;  category: ;  category: ;  category: ;

[pmuellr]

Left some notes on the implementation, and request for some additional tests.

[doakalexi]

When I follow your verification steps, I do see that this is working but when I tried to create an ES query rule agains an alerts as data index, I'm not seeing it work.

Thanks for catching this @ymao1, I was overwriting values and that is why you weren't seeing them. Should be resolved here 00406ed

[pmuellr]

LGTM!

[ymao1]

LGTM! Nice job :)

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1767488

Failed CI Steps

• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution

Test Failures

• [job] [logs] Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution / Enrichment Custom query rule Should has enrichment fields

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	57	63	+6
osquery	103	108	+5
securitySolution	439	443	+4
total			+17

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	65	71	+6
osquery	104	110	+6
securitySolution	516	520	+4
total			+18

History

• 💚 Build #83529 succeeded 8c6ce5c
• 💚 Build #83238 succeeded 1b28260
• 💚 Build #82901 succeeded 487f686
• 💚 Build #82810 succeeded 4b1578f
• 💚 Build #82101 succeeded 08dda95
• 💛 Build #81868 was flaky f689922

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream