[alerts] provide mustache functions for ease-of-use in transforming mustache variables

[pmuellr]

One feature of mustache that we are not currently taking advantage of is "functions", also sometimes referred to as lambdas.

It's not clear how much value we can add with these, but seems like worthy of investigation.

One of the use cases I have in mind is date formatting. We now provide a {{date}} variable in the mustache variables applied to action parameters, in ISO format. How might you provide a function which can format that date with a traditional date formatter? The challenge is, the arguments have to be in the "body" of the function invocation.

Here's an example of how this might work:

{#date-format}}{{date}} MM/DD/YY{{/date-format}}

The date-format function would be provided by us, and it's "arguments" would be <iso-date> MM/DD/YY. So the function would parse the arguments expecting a date as the first part, and the format as the second.

So many issues tho.

Presumably dates coming out of ES that are not under our control could be in any date format, so it's not clear how we'd parse; maybe we need a better separator like <iso-date> --- MM/DD/YY.

Ideally we could support various sorts of "string functions", and so dealing with some kind of separator story that allows strings to be transformed and additional arguments seems really hard/yucky. I think maybe we'd need to go with something #<iso-date>#MM-DD-YY# where the arg delimiter is passed as the first char, kinda thing.

We'd likely need a specialized "test" environment for this - the current action tester (beside the action editor) doesn't do any mustache templating, and shouldn't, because it doesn't have access to alert data.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[n0othing]

It'd also be helpful if we supported {{#toJson}}my_map{{/toJson}} + {{#join}}my_iterable{{/join}} functions, similar to [1].

One example use case would be sending an alert's tags as a JSON array: { "tags": {{#toJson}}tags{{/toJson}} }

[1] elastic/elasticsearch#18856

[pmuellr]

@n0othing - there is already some implicit support for array joins and json-ification. When you use an array object as a bare mustache variable, it will be rendered as if you had done arrayValues.join(','). If you use a non-primitive object (eg, {a: 42} as a bare mustache variable, it should render as if you had done JSON.stringify(objectValue).

[pmuellr]

Realizing that I haven't provided a reference to the existing mustache "functions" supported by elasticsearch - they seem to be documented here: https://www.elastic.co/guide/en/elasticsearch/reference/7.13/search-template.html

Quick take on them:

• "set default values" - {{my-var}} - I'm not sure this is really needed, as I believe you can do this with inverted sections
• "url encode strings" - {{#url}} - we already have some support for this, to renderer encoded depending on the whether the connector requires it. It's possible our support has some edge cases where you would need explicit encoding, so it's possible this might be needed. But also may be confusing if a user thinks they need it, but we are already doing it, then yielding double-encoded strings.
• "concatenate value" - {{#join}} - we already implicitly support joining arrays, but don't support the delimiters
• "convert to JSON" - {{#toJson}} - we already implicitly support this; mustache variables that are themselves non-primitive, non-array objects, are rendered as JSON.stringify(variable).
• "conditions" - I think this is just default mustache behavior

[scottdfedorov]

@n0othing - there is already some implicit support for array joins and json-ification. When you use an array object as a bare mustache variable, it will be rendered as if you had done arrayValues.join(','). If you use a non-primitive object (eg, {a: 42} as a bare mustache variable, it should render as if you had done JSON.stringify(objectValue).

The array field will join the entires with commas, however the problem with this is, it does not become valid JSON.

For example, if given the following json in a Kibana Alert action,

{
    "priority": "P3",
    "message": "{{alertName}}",
    "description": """ 
Some Description
    """,
    "tags": ["{{tags}}"]
}

Where "tags" is an array containing three tags, then what is output is the following:

{
    "priority": "P3",
    "message": "AlertNameHere",
    "description": """ 
Some Description
    """,
    "tags": ["TAG1, TAG2, TAG3"]
}

Which is not a valid JSON Array. There needs to be quotes around each of them.

What we need in the tags field is "tags":["TAG1", "TAG2", "TAG3"].

[pmuellr]

Yes, I think it's not possible, even with our {{.}} support, to do what you want with those tags.

I think you'd want to use the elasticsearch form of {{#toJson}}tags{{/toJson}} here?

Thanks for the use case!

[ehausig]

It would be a big timesaver for me to have the ability to dynamically generate links to specific monitors in Uptime, rather than having to put static links in each alert configuration. In Kibana 7.14.2, the state.monitorId exists however Uptime's URL path for linking directly to monitors requires the base64-encoded value of state.monitorId. The format appears to be:

{{kibanaBaseUrl}}/app/uptime/monitor/<base64-encoded state.monitorId>

While having a dedicated variable (like state.b64MonitorId) would solve my immediate challenge, I think building an expression that transforms the value could provide more flexibility in the long run.

Thinking about similar stuff I've seen in the wild, Helm template functions are kind of nice for transformations. For example:

{{kibanaBaseUrl}}/app/uptime/monitor/{{ b64enc state.monitorId }}

[cnasikas]

I think it will be very helpful to have:

Sha256

This function is useful when you want to create grouping keys to be used in external services like PagerDuty (dedup key) or ServiceNow (correlation ID) where there is a limit on the number of characters you can post.

{#sha256}}{{rule.id}}:{{alert.id}}:{{value}}{{/sha256}}

Nice to have:

Set

If you want to take the unique values of an array of objects. For example all unique hosts.

{#set}}{{context.alerts}} host.name{{/set}}

Sort

{#sort}}{{context.alerts}} host.name{{/sort}}

Filter

{#filter}}{{context.alerts}} predicate{{/filter}}

[pmuellr]

One thing we should do before adding these, is do a survey of other mustache usage within the stack - watcher and something else in Kibana (canvas?) are using "mustache", though I think handlebars is one of those variants in use. It would be good to align with other mustache "lambda" things that they have added.

[acrewdson]

We have also seens situations where it would be useful to be able to do simple conversions of values using a function in the template. Example: we have a context variable that represents latency in microseconds, but we'd like to display it in milliseconds when rendering the template.

[pmuellr]

For the use case above, it would be nice to provide math functions. We have an internal package available called @kbn/tinymath, which provides a large number of math functions, parser, and executor. Perhaps it would work something like this:

{{#Math}} {{context.someNumber}} / 1000 {{/Math}}

Where we would expect the context variables in the middle to be expanded out, and then a special Mustache variable #Math which is actually a function, would end up invoking TinyMath on it. Kinda based on the code in this experimental PR: #107612 - in this case, similar to the #Date variable.

This may not work well in practice. If a connector parm needs to be a number, we may not accept a mustache template as it's value currently, since that would be a string. I'm thinking both runtime validation within the connector type, as well as the existing UX.

[pmuellr]

Just had a thought on the problem here that seems to come up the most. Folks wanting to "pass through" a JSON array value as JSON. This doesn't work since we preserve array's toString() method, which is basically join(','), so that ['a', 'b', 'c'] is rendered as a,b,c.

Since we only support typical array access, there won't be any other properties on arrays that you could reference. For instance, while this is valid Javascript

const a = [1,2,3]
a.foo = 'bar'
console.log(a.foo) // prints "bar"

we end up cloning the context and would end up removing foo property on the array (not remove really, just not copy it over when cloning). That means ... we can add a new property after the clone, since there couldn't exist one before. Basically, iterate over the values in the context, and for any arrays, add a method asJSON to it, which will return the JSON representation. I think this would be in the same place we add toString() on objects:

kibana/x-pack/plugins/actions/server/lib/mustache_renderer.ts

Lines 64 to 78 in c4b43f9

	function addToStringDeep(object: unknown): void {
	// for objects, add a toString method, and then walk
	if (isNonNullObject(object)) {
	if (!object.hasOwnProperty('toString')) {
	object.toString = () => JSON.stringify(object);
	}
	Object.values(object).forEach((value) => addToStringDeep(value));
	}
	// walk arrays, but don't add a toString() as mustache already does something
	if (Array.isArray(object)) {
	object.forEach((element) => addToStringDeep(element));
	return;
	}
	}

So if context.a is the array ['a', 'b', 'c'], {{context.a}} would render a,b,c, but {{context.a.asJSON}} would render as ["a","b","c"].

[pmuellr]

Just had a user report an issue with url encoding of spaces within markdown links. To repro, create an rule that you can trigger to become active, and add an email action. The name of the rule should have a space in it, for instance "my rule". In the action, use the following template:

 [{{alertName}}](https://www.example.com?name={{alertName}})

You would hope this would get rendered as one of:

 <a href="https://www.example.com?name=my rule">my rule</a>
 <a href="https://www.example.com?name=my+rule">my rule</a>
 <a href="https://www.example.com?name=my%20rule">my rule</a>

The bottom two use an encoding for the space char; the hope would be that markdown adds this. The first just uses the space in the url as-is, in which case the hope is the browser will handle it - Chrome does today by changing the space to %20 when traversing the link.

However, what we get instead is this:

[my rule](<a href="https://www.example.com?name="my">https://www.example.com?name="my</a> rule

One way to fix this would be to provide some kind of mechanism - a lambda perhaps - to URL encode a value. So you might do something like this:

 [{{alertName}}]({{#URLencode}}https://www.example.com?name={{alertName}}{{/URLencode}})

[Danouchka]

• please, need something like {{format(mydate,"Europe/Paris")}} to write a date in a local timezone , or any other timezone.
  Does not make sense to send kibana alerts with time written in UTC. Thanks

[pmuellr]

Thought I'd note for folks watching: we've merged some mustache extensions slated (but not guaranteed!) to be shipped in 8.8.0; should help with date formatting, constructing JSON documents, and some math expressions.

adds mustache lambdas and array.asJSON #150572

[pmuellr]

Thought I'd note for folks watching: we've merged a mustache extension slated (but not guaranteed!) to be shipped in 8.9.0 for formatting numbers with Intl.NumberFormat.

adds FormatNumber mustache lambda #159644

[NybbleHub]

Hello all,

I'm currently using the {{#ParseHjson}}{{/ParseHjson}} Mustache lambda and I'm experiencing an issue with value containing the "&" character.

Here is an example:

{{#ParseHjson}}
{
    "alerts": [
        {{#context.alerts}}
        {
                "observer": {
                    "egress.zone":            "{{observer.egress.zone}}"
                    "ingress.interface.name": "{{observer.ingress.interface.name}}"
                    "ingress.zone":           "{{observer.ingress.zone}}"
                    "name":                   "{{observer.name}}"
                    "product":                "{{observer.product.asJSON}}"
                    "type":                   "{{observer.type}}"
                    "vendor":                 "{{observer.vendor}}"
                }
        }
        {{/context.alerts}}
    ]
}
{{/ParseHjson}} 

With the following value in the "observer.product" fields VPN-1 & FireWall-1

I'm getting the following error :error rendering mustache template "{{#ParseHjson}} and the result JSON is cut on the "&" character from observer.product value.

I've tried to add ".asJSON" at the end but it's not working.

Is there another lambdas like "asString" that I can use to escape special characters ?

Thanks,
Sébastien

[MakoWish]

Any chance to implement some sort of URL-encoding function? We get alerts from Detection Rules that often include usernames with spaces in them; Network Service or Local System for instance. If I want to create a Markdown URL to a dashboard, filtered by the user.name in the triggering event, the space completely breaks the URL.

[pmuellr]

I'm currently using the {{#ParseHjson}}{{/ParseHjson}} Mustache lambda and I'm experiencing an issue with value containing the "&" character.
...
With the following value in the "observer.product" fields VPN-1 & FireWall-1

I'm getting the following error :error rendering mustache template "{{#ParseHjson}} and the result JSON is cut on the "&" character from observer.product value.

I suspect there may be an escaping issue here. Can you try using the following - triple vs double quotes. This should bypass the escaping that may be occurring.

"product":  "{{{observer.product}}}"

[pmuellr]

I'm going to close this long-lived issue as we've added some support in this area in the last few releases:

• [ResponseOps] adds mustache lambdas and array.asJSON #150572
• [ResponseOps] adds FormatNumber mustache lambda #159644

Beyond that support, it seems like there are three other areas of interest, which we either had an issue open for, or I just opened one:

• string encoding: [ResponseOps] add mustache string encoding support #168728
• string hashing: [ResponseOps][Alerting] Dedup key limit #142757
• array functions: [ResponseOps] add mustache array function support #168729