[Logs UI] [Design] Machine learning analysis screens

[jasonrhodes]

For the planned Logs/ML integration to enable ML categorization on logs, we need designs for a number of screens.

AC (Acceptance Criteria, or the design deliverables we need when this ticket is done):

• Screen: enable ML analysis, onboarding
  • Possible link to article about ML categorization
  • Optional field for selecting ECS event.dataset filter
  • Field to categorize input may be a dropdown of text-type fields in their log index

---

• Screen: View ML analysis in progress (charts etc)
  • We are opting for the SIEM tab style for now, to be re-considered across all observability at a later time
  • No query/filter bar in this view (not possible to filter ML data)
  • This view should show the existing config somehow (field being analyzed, any ECS dataset filter applied, etc.) --> these cannot be edited but should be shown for context
  • Severity score is 0-100 ML score that represents relative intensity of the anomaly (this log is extremely rare, this category is appearing much more frequently than it usually does, we should confirm this 100%) --> how do we be clear about this?
  • Need to understand whether sorting is possible/useful
  • We can ignore the "Log rate severity" on the top graph of this mockup for now

---

• Screen: View current settings/disable/remove ML analysis (probably should live in Settings screen?)
  • Can't edit these, can only remove and re-create, which causes you to lose access to your old categories since we would only query against the new job from that point forward (the data would still be there but the UI would ignore it)
  • I assume we still need to allow users to turn this feature off if they enable it, though

[elasticmachine]

Pinging @elastic/infra-logs-ui

[hbharding]

Figma link: https://www.figma.com/file/T08kdh6YQcVCx1tzljqNHF/Logs-ML-Integration?node-id=0%3A1

[weltenwort]

After studying the ML result data structures and talking to people more knowledgeable about them, there seems to be a mismatch between the implied features in the Mockup and what is feasible using the output of the ML jobs. In particular, the association between the categories and the time filter is only possible via an anomaly record. This has a few consequences:

Log Categories: What shows up as a category here are occurences of categories in specific time buckets that behave anomalously. This means that a category might show up multiple times.

Log Categories Count column: The count is the anomalous document count within a specific bucket. Any occurences of documents belonging to a category, which doesn't exhibit anomalous behavior in the specific bucket will not be included. Since there is no link between the documents and the categories beyond the few example term queries, we can not query for the overall count of documents belonging to a category.

Log Categories Count sparkline: The sparkline is an expansion of the count over time. The same limitations of the previous point apply.

It appears the prototype that inspired this mockup was based on a custom build of the ML plugin that produced different output data than what is currently produced.

[Zacqary]

EUI doesn't seem to have built-in stuff to do the type of padding and layout that's in these mockups. Do we want to do a bunch of custom styling to get all that whitespace, get the "More data" help text next to the "Time range" field, etc.? Or just get as close as we can with standard EUI components?