[Security Solution][Response Ops] Rule actions sending malformed JSON when using {{context.alerts}} action template

[dplumlee]

Rule actions are failing to fire when using the {{context.alerts}} placeholder due to the JSON output being malformed when sent to the assigned webhook. This is causing 502 errors to come back from the webhook and not properly fire when the rule executes successfully. When looking at the malformed output, the JSON is often arbitrarily cut off and not escaped properly which is causing it to error out.

So far the action template we've used in testing has been

{\r\n   \"rule.name\" : \"{{rule.name}}\",\r\n   \"description\" : \"{{context.rule.description}}\",\r\n   \"rule.severity\" : \"{{context.rule.severity}}\", \r\n   \"alert.ID\" : \"{{alert.id}}\", \r\n   \"context.alerts\" : \"{{context.alerts}}\", \r\n   \"signal.tags\" : \"{{rule.tags}}\",\r\n   \"rule.id\" : \"{{rule.id}}\"\r\n}"

but we've also modified it with different forms of mustache syntax in the template and gotten similar, malformed results. This has also been tested on multiple different webhooks each with similar 502 outputs. The issue seems to consistently be occurring with the {{context.alerts}} placeholder, but it's possible the issue is not limited to that field alone.

In the SDH (internal link) this was originally discovered in, the same action template and rule action was working for a certain rule while failing for others, leading us to believe maybe it has something to do with the alerts themselves not meshing well with the mustache formatting or something to that effect.

We've triaged it to the formatAlertsForNotificationActions sending alerts to the alerting framework in this file as still working and sending valid JSON so presumably the bug is downstream from that.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[pmuellr]

When looking at the malformed output

Could you provide some sample output here?

The issue seems to consistently be occurring with the {{context.alerts}} placeholder

What is the value in context.alerts? If it's an array, we got problems. We don't currently have a way of expanding out an array like this, as JSON, in the rendered output.

So far the action template we've used in testing has been {{elided}} but we've also modified it with different forms of mustache syntax in the template and gotten similar, malformed results.

I haven't seen anyone successfully render an array of JSON values as a valid JSON value in the output, but I'd be interested to see what you try.

The most relevant issue related to this is this one: #84217 - just had a quick thought on a low-cost way of adding better JSON support for arrays, and posted a comment there.

[akusei]

I'm having this same issue and I've tried many different iterations of mustache templates to get the output to be valid json. The weird think here is that the email connector parses the mustache template properly and produces the expected result.

{
  "raw_json": [
  {{#context.alerts}}
    {{.}},
  {{/context.alerts}}
    {}
  ]
}

The above renders as "{\r\n \"json_raw\": [\r\n{<alertdoc1>},\r\n{<alertdoc2>},\r\n{}\r\n]\r\n"
What's interesting though is the email connector does not have this issue. It will happily take the above template and produce valid json. I've also tried these which produces the same results

{
  "raw_json": [{{context.alerts}}]
}

[
{{#context.alerts}}
  {
    "raw_json": {{.}}
  },
{{/contexst.alerts}}
  {}
]

This is really difficult to work with, especially since the elastic alerts can and for us most times will have multiple alerts per execution. A workaround I'm using for now is to render the json as follows and then parse the raw_json value on the remote end with a call to python's json.loads

{
  "raw_json": "[{{#context.alerts}}{{.}},{{/context.alerts}},{}]"
}

[jamesspi]

I thought I'd mention that some users ran into a similar issue when not explicitly setting the content-type header to JSON. There was a 7.17 update that changed the underling axios version, and it was defaulting to a content-type of form.

[aarju]

We use the webhook connector to send our alerts to our SOAR system internally. I responded to this discuss post with how we do it. https://discuss.elastic.co/t/elastic-rule-connector-sends-a-string-instead-of-json-to-the-webhook/313833

We send the entire contents of the alert body as an ndjson with this action template:

{{#context.alerts}}
{{{.}}}
{{/context.alerts}}

[ersin-erdal]

Hi,

This issue came up as an sdh issue as well.

Unfortunately there isn't built-in way in mustache to tackle this, but IMHO, we can do:

1- Create an issue to add an extra field (like {last:true}) to context.alerts items, so we can detect the last item in the list and use inverted sections feature of mustasche to skip rendering the last comma.
e.g.: {{context.alerts}} ... {{^last}}, {{/last}} {{/context.alerts}}

This has to be approved by the @elastic/security-solution as they are the owner of the rule type.

2- We can try to replace trailing commas by using a regex in the webhook connector type executor.
Sounds a bit clumsy to me though.

kibana/x-pack/plugins/stack_connectors/server/connector_types/webhook/index.ts

Line 199 in d65b02c

const { body: data } = params;

cc: @mikecote @XavierM