[Actions] ES Index action should be able to iterate over context variables

[ymao1]

The ES Index action currently asks users to create a single document to index each time the action runs. This action can incorporate context variables using mustache templating. It would be useful to allow the Index action to use mustache templating to iterate over a context variable that is an array and specify a document to index for each element of that array.

This was discussed specifically as part of the action handling for the search alert when the hits are passed as an action variable. If we have 1,000 hits for an execution of this alert, we should be able to bulk index 1,000 documents with a single ES index action.

As a reference, other connectors, like the slack connector, can do something like this:

You have {{context.value}} matches!
{{#context.hits}}
  Document with {{_id}} and hostname {{_source.host.name}} has {{_source.system.memory.actual.free}} bytes of memory free
{{/context.hits}}

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

Is the problem here that you can't use mustache sections at all? Which I don't think you can, because the documents param for the index action is spec'd to be a JSON object. Compare that to the webhook action, which also expects JSON, but specs the body param as a string.

That thought is referenced in issue comment, and I believe is spot-on correct: #78877 (comment) - we need to change the documents param to be a string, like the webhook body. How we get there (extend the type to allow strings, or convert to a string and migrate existing actions) is not clear to me.

So, these are kinda DUPs, but I think the referenced issue is kinda noisy and there's a lot in there about how to make JSON validation work still work in the UI (which seems pretty much impossible, once you throw in mustache sections for looping).

Perhaps we should re-target this one to just get the type of the documents param changed to a string, we can leave the other one open to noodle on how to do JSON validation in the UI ...

[ymao1]

The documents field I believe is already spec'd to be an array of objects:

kibana/x-pack/plugins/actions/server/builtin_action_types/es_index.ts

Lines 38 to 40 in 1ab03d0

	const ParamsSchema = schema.object({
	documents: schema.arrayOf(schema.recordOf(schema.string(), schema.any())),
	});

I believe the issue is with the UI editor validating for JSON objects only so if you try something like

{{#arrayContextVariable}}
{
  "_id": {{_source._id}}
}
{{/arrayContextVariable}}

it gives you a validation error. The linked comment makes it seem like a more complicated issue though so it sounds like I'm missing something :)

[pmuellr]

Yeah, sorry, it's kinda confusing.

es index spec's it's documents param as objects:

kibana/x-pack/plugins/actions/server/builtin_action_types/es_index.ts

Lines 38 to 40 in 7609fb9

	const ParamsSchema = schema.object({
	documents: schema.arrayOf(schema.recordOf(schema.string(), schema.any())),
	});

webhook spec's it's body param as a string:

kibana/x-pack/plugins/actions/server/builtin_action_types/webhook.ts

Lines 70 to 72 in 7609fb9

	const ParamsSchema = schema.object({
	body: schema.maybe(schema.string()),
	});

They are both intended to be JSON (actually, webhook body could be XML, plain text, etc - part of the reason we didn't spec it as a JSON-able object).

The string version seems like what we should have done with es index, because there's no way to represent mustache templates with "sections" (eg, for looping) as JSON. As a string though, sure!

a bit more background:

The further complication is ... how do we validate this mustache-infused JSON as JSON - particularly in the UX. I suspect we can't. There are other things you can't do like use an "unquoted" mustache variable, which would work perfectly fine at runtime (hehe - even GH marks the double-braced variable reference as an error!):

{
   "foo": {{context.variable-that-is-a-number}}
}

There are probably a lot of cases where a customer wouldn't use mustache sections, or need to use unquoted mustache variables like above, that could end up validating as valid JSON, so there are cases where the validation is useful.

Not sure what the answer to this "what do we do about UX validation for these" might be. Allow validation by default, but let the user turn it off if it gets annoying (eg w/ a checkbox)? Just stop validating in the UX altogether? Hope we could figure out some way of actually doing validation with the mustache {{...}} bits mixed in?

Presumably we'd always be doing runtime validation on such strings, once rendered into their final form via the mustache rendering, but that's obviously later than we'd want - when the action is actually executed. I suspect that's where we'll end up though, in terms of the only place that's safe to validate the documents are valid JSON.

In any case, we can't ever do things like mustache sections with es index documents param, till we change that param to allow strings, so that we can get the action type validation to not fail. And then once we allow strings, we'll have the same UX validation errors for mustache-y bits, that webhook has today.

[ymao1]

Gotcha. Thanks for the explanation!

[pmuellr]

Having written all that, I'm not sure we have an explicit issue open to migrate the documents param in es index from an object to a string, but I think there's no way around it - we have to.

I'm not seeing an issue, maybe @mikecote remembers one? If we don't already have one, perhaps we could treat this issue as a meta of how to get this fixed, which would be

1. change the mapping, add a migration, fix the server and UI code to deal with a string version
2. figure out what to do about the JSON validation of this in the UI (webhook has similar issues I think - it will complain about mustache-y bits that make the body not JSON parseable).

[mikecote]

I'm not seeing an issue, maybe @mikecote remembers one?

@pmuellr Not that I can recall. I did a quick search as well and couldn't find anything.

[juliettejl]

I see this is quite an old thread, but this functionality would be very useful if there is a way around the validation issue.
Currently for a project I am working on we are having to re-format the documents via a logstash pipeline to get around this but this is a fiddly process and it would be far more convenient if there was a way to do this via the Index Connector.