[Logs UI] [Epic] Converge Logs UI and Discover UX

[smith]

🗒 Summary

Elastic's goal is to have the best logs experience. This issue proposes to turn the Logs UI into a "Log Explorer" that is composed of components shared with Discover but offers additional features tailored for log consumption.

🪨 The current situation

When viewing logs in Kibana, users have a few options:

• The Discover document explorer allows you to view a list of documents
• The Observability logs UI views log entries in a console tail-like experience
• Log streams can be embedded in dashboards
• Log streams are embedded in APM service and trace views

The document explorer is used in Discover. The log streams are provided by the LogStream embeddable component.

Each of these ways of viewing logs has advantages and disadvantages.

🏁 Primary goals of this effort

• 🔵 Align UX with Discover
• 🔵 Provide a more feature-rich embeddable log explorer

🔍 Feature comparison

🟪 Current document explorer	🔶 Current Logs UI	New Log Explorer
Unified search bar with filtering	Simplified KQL bar search	🟪 Unified search bar with filtering
Unified search data view selection on search bar	Log source selection under Settings	🟪 Unified search data view selection on search bar
On-page column and sorting customization	Column customization on Settings page	🟪 Discover field selector and sorting customization
Inspector has table and JSON views	Inspector only has table view	🟪 Discover fly-out
Multiple document selection	Single document selection	🟪 Multiple document selection
Fixed set of documents with pagination	Infinite scrolling	🔶 Infinite scrolling
Auto-refresh	Stream live	🔶 Stream live
Only usable in Discover app	Embeddable in dashboards and apps	🔶 React component and Embeddable
View surrounding documents	View in context of infrastructure (host, container)	🔶 View in context?
Horizontal date histogram	Vertical date histogram	🔶 Vertical lens-based date histogram
Support indices with or without timestamp	Only support indices with timestamp	🔶 Only support indices with timestamp
Store settings in Saved Search	Store settings in Log View	🟪🔶 Store settings in Saved Search, support reading Log Views

🗺️ Implementation Plan

Phase 1: Exploration

Do some prototyping to figure out what we would need to put logs in Discover.

Tasks

• [Logs UI] [R&D] Investigate EuiDataGrid for log stream #79288
• [Logs UI] [R&D] Prototype Log Explorer mode for Discover #134764

elastic/eui#5943 has some further discussion about multiple ways of presenting data in an EUI data grid.

Phase 2: Unified Search in the Logs UI

The Logs UI uses the same unified search bar component like Discover. It therefore has the same query, filter and data view selection UX as Discover.

Tasks

(order derived from the dependency analysis at the end)

• [Logs UI] Use Unified Search Bar: support query input #142719
• [Logs UI] Use Unified Search Bar: support date picker #142767
• [Logs UI] Analyze page and URL state #142775
• [Logs UI] Refactor log stream page state #145131
• [Logs UI] Refactor log stream page query state #145133
• [Logs UI] Refactor log stream page time state #145137
• [Logs UI] Add log stream page filter state and UI #142764
• [Logs UI] Support inline log views in http routes #151489
• [Logs UI] Support inline log views in the UI components #142840
• [Logs UI] Enable loading and saving of unified search queries #148310
• [LogsUI] Use Unified Search Bar: support DataView selection #142847
• make query and filter available in the shared React component

Later: Convergence of the field selection UX

(the scheduling of this and the other "later" phases depend on the roadmap of the Discover team and possibly other Kibana platform teams)

The Logs UI offers the same field/column selection UX as Discover.

Tasks

• extract field selector from Discover
• use Discover field selector

Later: Convergence of the document fly-out UX

(the scheduling of this and the other "later" phases depend on the roadmap of the Discover team and possibly other Kibana platform teams)

The Logs UI uses the same fly-out as Discover.

Tasks

• extract fly-out from Discover
• use Discover fly-out in the Logs UI
• use Discover fly-out in the shared React component

Later: Convergence of the grid rendering

(the scheduling of this and the other "later" phases depend on the roadmap of the Discover team and possibly other Kibana platform teams)

The Logs UI uses the same grid component as Discover, but with bi-directional infinite scrolling and live streaming.

Tasks

• load log entries and summary directly
• [Logs UI] Identify critical features of EuiDataGrid suited for dynamic data #148417
• create suitable EuiDataGrid variant
• use data grid
• rethink "view in context" UX

Later: Integrate closely with Discover

(the scheduling of this and the other "later" phases depend on the roadmap of the Discover team and possibly other Kibana platform teams)

Tasks

• support switching between Discover and Log Explorer

Dependencies

flowchart LR

uxAlignment[align UX with Discover]
betterEmbeddable[better log explorer embeddable]
unifiedSearchBar[use unified search bar with filtering]
unifiedSearchDataView[data view selection on search bar]
refactorState[refactor Logs UI state]
settingsCrud[move settings CRUD to stream page]
transientSettings[support transient settings]
loadDataDirectly[load data directly]
columnCustomization[column customization]
discoverFieldSelector[use Discover field selector]
extractFieldSelector[extract field selector from Discover]
discoverSorting[use Discover sorting customization]
discoverDataGrid[use data grid]
extractDataGrid[extract data grid from Discover]
discoverFlyout[use Discover fly-out]
extractFlyout[extract fly-out from Discover]
documentSelection[multiple document selection]
fullReactComponent[wrap the full Log Explorer into a React component]
fullEmbeddable[wrap the full Log Explorer into an embeddable]
switchDiscover[support switching between Discover and Log Explorer]
loadSavedSearch[support saved searches as log views]

uxAlignment --> unifiedSearchBar
uxAlignment --> unifiedSearchDataView
uxAlignment --> discoverDataGrid
uxAlignment --> discoverFlyout
uxAlignment --> columnCustomization
uxAlignment --> documentSelection
uxAlignment --> switchDiscover
unifiedSearchBar --> refactorState
unifiedSearchDataView --> unifiedSearchBar
unifiedSearchDataView --> transientSettings
transientSettings --> settingsCrud
columnCustomization --> discoverFieldSelector
discoverFieldSelector --> extractFieldSelector
settingsCrud --> refactorState
columnCustomization --> discoverSorting
discoverSorting --> discoverDataGrid
discoverDataGrid --> refactorState
discoverDataGrid --> extractDataGrid
discoverFlyout --> extractFlyout
documentSelection --> discoverDataGrid
documentSelection --> refactorState
switchDiscover --> loadDataDirectly
switchDiscover --> loadSavedSearch
loadSavedSearch --> transientSettings

betterEmbeddable --> fullReactComponent
betterEmbeddable --> fullEmbeddable
fullEmbeddable --> fullReactComponent
fullReactComponent --> loadDataDirectly
fullReactComponent --> unifiedSearchBar
fullReactComponent --> transientSettings

classDef default stroke:#ff9900,stroke-width:2px;
classDef primaryGoal stroke:#1277d3,stroke-width:2px;
classDef kibanaTeamDependency stroke:#ac45bd,stroke-width:2px;
class uxAlignment,betterEmbeddable primaryGoal
class extractFieldSelector,extractDataGrid,extractFlyout kibanaTeamDependency

Loading

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[ruflin]

I like that you split up Document and Log explorer. There are overlaps between the two products but also enough differences. One key part for me is that we are smart about when we should which one. For example any user exploring logs-* indices should automatically get the Logs explorer view.

Even though the two components likely can share lots of code, I would not make this the driving factor. Lets ensure that we don't make tradeoffs on the UX and functionality just to share more code. We can always generalise at a later point.

[kertal]

Thx a lot for this comparison! One note about Document explorer Only usable in Discover app, when you save a search in Discover, you can add it to a dashboard, so document explorer is available as embeddable.

[weltenwort]

@smith this is a great summary, thank you

There are some additional difference in the status quo:

Document explorer	Log Explorer
indices selected using a data view	indices selected using a name pattern or data view
cell content based on fields or runtime fields	cell content based on fields, runtime fields or reconstructed message
arbitrary sorting	always ascending time-based sorting (like a log file)

Whether we want to carry these forward depends on how backwards-compatible we want to be.

[ghudgins]

indices selected using a name pattern or data view

small note - platform team is looking to offer this in underlying embeddables via "ad hoc data views" (or "by value" data views) soon - #126911

[smith]

@kertal @weltenwort @ghudgins thanks for the feedback. I've updated the tables.

[smith]

Updated description to more closely match what we're doing.

[smith]

This work has not all been completed as specified above, but we're doing other work to support these goals. Closing this and we'll open another issue for follow-up if needed.

[melkamar]

Could you link to the issue that tracks the Logs UI improvements please, when this one has been closed? I can't find anything and the logs improvements did not appear in 8.8. Thanks!

[smith]

@melkamar this is probably the most useful public issue currently: #164197.

The Log Explorer UI is expected to be Beta in 8.11, though things could change.

Here's a screenshot of a pre-release build:

[melkamar]

Nice, thanks for the update!