[Infra UI] [Epic] Use-case specific logs ui views

[weltenwort]

📓 Summary

We want to improve the way the Logs and Metrics UI, their respective alerts, components, and Embeddables are configured to allow for more robust, performant and flexible usage. This epic wants to improve both the UX and the technical basis using several goals as guideposts.

🎯 Immediate goals

Goal 1: Improve user awareness of configuration in the Log stream page

The "Settings" tab of the Logs and Metrics UIs are not easy to find via their button in the top right navigation bar. Many users don't seem to be aware that they can change any settings to influence the which data are displayed in the respective apps. They are even less often aware that these settings influence the behavior of rules owned by the apps.

Instead, users should be made aware that the settings exist and what effect it might have to change them.

Goal 2: Reduce implicit coupling between the Log stream page and the alerting rules (robustness and performance)

As mentioned in goal 1, the rules owned by the Logs and Metrics apps use the indices configured in the source configuration to determine which indices to query. This is mentioned in the docs but can't be deduced by looking at the UI. The implicit coupling can cause surprising behavior of rules and even excludes some use-cases. The workarounds that users apply often have a negative impact on the performance as the set of matched indices tends to grow over time.

Instead, the user should be able to tell and customize which source configuration a rule uses. It should be possible to create rules targeting different indices without interfering with each other or the normal usage of the UIs.

Goal 3: Reduce implicit coupling between the Log stream page and Log Stream embeddable component (increased flexibility)

The log stream component and Embeddable currently take a source id to specify the source configuration used for data access. That means any change to that source configuration might inadvertently influence an embedded component elsewhere unless the embedding application has taken care to inject their own configuration. At the same time, a dedicated configuration by an embedding application can't be easily navigated to and customized by the user.

Goal 4: Enable multiple use case based views in the same space

Many users would like to narrow the logs and metrics accessed by the UIs so they can reflect different focused use-cases within their organization.

Goal 5: Reduce implicit coupling between Metrics UI and Logs UI (UX, less confusing for user)

A single saved object is currently used to store the per-space settings for both the Logs and Metrics UIs. We've seen situation in which this has caused problems with the indices of one app to break the other app. This is confusing to the user. At the same time it prevents us from offering guided setup workflows for both apps without one influencing the other.

💡 Critical path to “multiple saved views” (in order)

• [Logs UI] Use dedicated configuration saved object #120924
• [Logs UI] [WIP] Prompt for a data view when log source not configured #98277
• [Logs UI] Explicitly reference a LogsViewConfiguration in log threshold rule parameters #120928
• [Logs UI] Allow for internal configurations to be overridden using a saved object #120929
• [Logs UI] Make configuration ID required in Log Stream component #120930
• [Logs UI] Inject internal configuration per consuming app #120931
• [Logs UI] Allow/require user to choose which configuration to use #120932

🚀 Future goals

• Enable Agent based integrations to ship domain curated configuration (#120803)
• Enable in context configuration (User local views, stored in query or localstorage, runtime fields) ([Logs UI] Move Log stream settings onto the Log stream page #124011)

Related notes

Make Log Stream component and APIs accept inline fallback configuration [3] 
[To use until the user saves a real config under the id chosen by the embedding app]
[pull configuration resolving to the very beginning, big]
(Optional) Export LEGACY!!!!  Fallback inline LogsViewConfiguration for convenience [3]

Allow user to create/delete new LogsViewConfiguration [4]

Extend LogsViewConfiguration with filter, time range (with live stream setting) and UI settings (font size, line breaks) and update Settings screen [4]

Allow users to select which LogsViewConfiguration to use for a Rule (maybe by cloning) [2]

Surface connection between LogsViewConfiguration and Rules under Settings screen [1,2]
Surface connection between LogsViewConfiguration and ML jobs under Settings screen [1]

Add LogsViewConfiguration ID to ML job custom_settings (assume “Default” for existing jobs) [1]
Update “ML sync check” code to use stored ID [1] (ID is included in name/id of the ML job)

Build LogsViewConfiguration selection/create new component [3]

Add further validation to the LogsViewConfiguration (such as, will all related Rules still work with this configuration, or the added index won’t add any document due to `canMatch`) [1,2]

Allow setting a default LogsViewConfiguration for a space

Automatically sync ML jobs based on LogsViewConfiguration (BIG maybe)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[smith]

We're not planning on completing this. Leaving individual issues opened.