[Logs UI] [R&D] Prototype Log Explorer mode for Discover

[Kerry350]

Summary

As part of #134412 we want to investigate the feasibility of having a Log Explorer mode in Discover.

"Discover log explorer" as a companion to the "Discover document explorer". The log and document viewing experiences will be built on the same underlying foundation to improve quality and consistency while reducing maintenance costs.

POC Goals

ℹ️ This list is not a strict set of acceptance criteria but an enumeration of features potentially worth demonstrating. We may decide to add additional features or skip listed features as the investigation progresses.

• There is a new sub-route in discover that renders a log explorer.
• The log explorer re-uses as many components as possible from the discover document explorer:
  • saved search controls at the top
  • query bar at the top
  • date picker at the top
  • field list to the left
  • the column selector
• The log explorer displays log entries using the <EuiDataGrid> such that...
  • the grid is initialized with a large number of virtual rows (10_000, for example)
  • the grid scrolling is limited to the area of loaded entries plus a loading indicator row at the top and the bottom (also depends on EUI 6.2.0)
• The state management is written such that...
  • it loads a subset of log entries to render in the middle of these virtual rows
  • it prepends and appends log entries to the chunks of loaded log entries (this might have to be manually triggered until EUI 6.2.0 includes onGridItemsRendered support)
  • it uses the query techniques we learned when building the Logs UI (including search_after with a unique tiebreaker, async search with cancellation)
• The log explorer contains a vertical date histogram to the right of the grid...
  • on which the currently visible time interval is highlighted
  • the boundaries of which are determined by the page's date range
  • which shows a breakdown of log entry counts by a field
  • clicks on which cause the data grid to load the log entries around the clicked-upon time
• When setting the date picker to auto-refresh with a relative time range that ends at "now", poll the latest log entries (live tailing mode)
• The log entry rows demonstrate log-specific features such as a cell popover that shows tracebacks associated with the log entry
• The log explorer contains a query inspector that shows the queries used on the page

🎨 Preview screenshots

Areas of investigation

⚠️ This is not exhaustive and further topics will likely arise once this issue is picked up. This aims to provide a high level overview at this stage.

Preliminary setup

• We will need to setup any changes behind a feature flag.

• There should be some sort of toggle to switch in to Log Explorer mode. There is no need for the UI / UX behind this to be perfect at this stage, as it will be hidden behind a flag. We can probably just pop it as an option here for now:

Visual investigation

EUI DataGrid

Investigations into enhancements for the DataGrid will be happening in #79288. We should consider how those API changes will interact with this new Log Explorer.

Histogram sidebar

E.g.:

• Are there any technical advancements (be it in EUI, native JS functionality etc) that we can take advantage of for the Logs histogram / minimap sidebar?

• How should this sidebar be integrated in to the Discover page?

  • Should the amount of space distributed between the log rows and the histogram be configurable, if so how?

Data investigation

• The Logs UI Log Stream currently uses search_after to incrementally load results (and a reversed search_after to achieve a "before" search). Discover does not use search_after currently. We should investigate the query and data usage:
  • Should we have a separate query mode for the Log Explorer that uses search_after, or should the two converge? (e.g. the document explorer would also use this new query mechanism)
  • What sorts of parameters are we dealing with in the current query flows? Is there anything missing that the Log Explorer would need?

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[weltenwort]

ℹ️ I updated the description with the current progress

[weltenwort]

ℹ️ Status update

⚙️ What works in the prototype

• main grid
• incremental loading
• vertical histogram with breakdown, click to jump
• flyout
• filters
• runtime fields
• field formats
• saved searches
• tailing
• columns functionality

🔮 The path forward

Scenario 1: Separate log explorer mode

• separate route
• re-uses grid + flyout
• re-uses some state/saved search logic
• separate embeddable

Advantages

• limited collision with ongoing Discover work (such as SQL support)
• can ignore all legacy functionality (i.e. the old docs table)
• only needs to support time-based data views

Disadvantages

• adds another parallel implementation for the app and embeddable
• we'll have to actively port future features from discover's "main" mode to the log explorer mode

Scenario 2: Refactor main discover state management

• three modes in the main route:
  • "plain record" (SQL)
  • "document" (classic discover)
  • "events" (time-based, starting with logs)
• refactoring of state management
  • no SQL special cases
  • state machine based
  • data fetching specialized for the respective mode
  • avoid separate saved search embeddable implementation
• query bar, grid and field list generalized for use by all modes

Advantages

• remove SQL special cases
• remove embeddable special state management
• predictable basis for future features
• less divergence due to unified state management and generic components

Disadvantages

• collision with other Discover efforts
• complex planning:
  • requires in-depth survey of existing functionality
  • incremental steps hard to carve out

High level diagram / overview

This is a high level (not 100% exhaustive) overview of the structure based on the prototype, and where we have sharability and divergences with Discover.

[matschaffer]

I saw this in the "In Review" column, so I figured I'd mention my ¥2.81 would be to go with scenario 2 since it avoids special cases and divergence.

Though I'm sure there's a conversation to be had with discover maintainers to see how we can weave that into other efforts.

[kertal]

I agree that scenario 2 would have plenty of advantages, but also like Felix mentions correctly:

• collision with other Discover efforts
• complex planning:
  • requires in-depth survey of existing functionality
• incremental steps hard to carve out

maybe we find a way to follow scenario 1 to tech preview (with increasing the common shared code) before we can go to scenario 2 which is a pretty big change in Discover

[kertal]

KUDOS for this great diagram, that I think @Kerry350 drew?

[Kerry350]

@matschaffer

Though I'm sure there's a conversation to be had with discover maintainers to see how we can weave that into other efforts.

Indeed, conversations are ongoing with the Discover folks.

maybe we find a way to follow scenario 1 to tech preview (with increasing the common shared code) before we can go to scenario 2 which is a pretty big change in Discover

This seems like a good compromise. My preference would also be scenario 2, but I appreciate there is a huge planning overhead. Pushing the POC to a preview state with no (that we know of 👀) bugs, and trying to share even more code seems like a fair next step. One thing to mention is that Felix and I were keen to stay involved, and not just dump this on the Discover team.

KUDOS for this great diagram, that I think @Kerry350 drew?

Thanks! Yeah, that was me. I'm sorry it wasn't 100% finished, I ran out of time before the demo. But I still have the excalidraw file if there's value in adding to it later.

[smith]

Done! 🚀