Truncate lastFailureMessage for siem-detection-engine-rule-status documents

[rudolf]

siem-detection-engine-rule-status documents stores the lastFailureMessage a string which is indexed as type: "text" but some failure messages are so large that these documents are up to 26MB.

These large documents cause migrations to fail because a batch of 1000 documents easily exceed Elasticsearch's http.max_content_length which defaults to 100mb.

Even though we're fixing the root problem with migrations in #107641 these large documents still consume a lot of resources like storage and particularly heap and in my tests easily cause Elasticsearch to run into circuit_breaking_exceptions.

Can we truncate these failure messages to limit the size of these documents and/or not index the entire contents of this field?

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

A note on why this can happen:

Every rule execution runs a loop under the hood. During each iteration of this loop Detection Engine queries source indices (containing source events), a small time period per iteration. If it finds any events matching the rule criteria, it creates detection alerts from them and attempts to bulk index them.

When querying and bulk indexing within an iteration, it’s possible that we catch exceptions. All the exceptions are being collected and returned from this loop. Then Detection Engine joins all of them (all their messages) and changes the status of the rule to failed with this Bulk Indexing of signals failed message. So it doesn't necessarily mean all errors there are indexing errors, it can contain search errors as well.

So when the loop contains a lot of iterations, it’s possible to get a very large list of caught exceptions and a long status message. So we definitely need to truncate this list.

[banderror]

A note on how to truncate:

1. Truncate on write:
   • get unique error messages from the list
   • truncate the list (leave max 10-20 errors, for example)
   • join its items to a string
   • truncate the resulting string (leave max 1024 characters, for example)
2. Write a migration for siem-detection-engine-rule-status that would truncate existing status documents:
   • truncate the existing message fields (leave max 1024 characters, for example)
3. Consider adjusting the mappings of siem-detection-engine-rule-status (e.g. using ignore_above, but I guess it's only for keywords?)

[banderror]

A note on the impact and which version to target from @rudolf:

This has a high impact on users and causes downtime for them and support escalations. Even if we are fixing some of the problems already this is also causing OOM errors for users with smaller 1GB Kibana instances which is harder to address. At the moment it doesn't seem to be affecting a large number of users, but I suspect this will increase as more users adopt these features.

This sounds like it might be a trivial fix, if that's the case I hope we can fix it by 7.14.2