[Security Solutions][Detection Engine] Critical bug where value lists were not operational

[FrankHassanabad]

Summary

Fixes bugs to allow users to use value based lists manually. This isn't a first class citizen of the UI at the moment but you can manually add them to the existing UI as long as it's a single index and does not mix ECS threat lists with item lists.

Example is upload a list in the file hosts.txt and a type of keyword:

Then add it as a threat mapping using:

• Index of .items-${space_id} such as .items-default
• Use the mapping field of "keyword"
• Use the query of list_id: ${file_name} such as list_id : "hosts.txt"

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: db6202a

Metrics [docs]

✅ unchanged

History

• 💚 Build #81393 succeeded d509e54
• 💚 Build #81361 succeeded c76d3c4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[rylnd]

I had one question about documentation, but this looks great! I created a rule as described on master, and saw the mapping error:

I then checked out this branch and the rule's next run succeeded 👍

[rylnd]

I initially created a rule against a threat index like .items-*, which does not satisfy this condition and thus still suffers from this issue. This is mentioned in the description as "single index" but I wanted to call it out again since it's easy to miss.

@FrankHassanabad do we want to add a note about this in the threat match rule documentation, or is too much of a "power user" scenario?

[FrankHassanabad]

I think it's a bit of a "power user" scenario and a bit of a trick. We can support it nicely in the UI because we can query for the index list item name and populate it like it is here and then disallow additional indexes as if you choose to use a plain list one that's all you can choose since value based lists aren't ECS standards.

For API users they will have to know their space names ahead of time and kind of fill in the API so we might want to update at least the API docs before release time to show an example of how to do it or we might need to add a better API for when they have a mapping and want to refer to a value based list.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)