[Log Explorer] Proposal for virtual column implementation

[weltenwort]

Summary

This is a proposal for the addition of a new kind of columns in the Log Explorer. It enumerates some use-cases and also proposes a means of implementation.

• 🔗 related to: [Logs UI] Add "node" name as column option to stream #45477
• 🔗 related to: [Logs Explorer] What content to show if message field is empty #166138
• 🔗 related to: [Log Explorer] Show log.level by default in table (if available) #166680
• 🔗 related to: https://github.com/elastic/logs-dev/issues/43

Use cases

Resource context column

In addition to the timestamp and message, log entry documents usually contain a large set of fields that contain metadata about the resource this log entry originated from or is related to. This includes the host name, container ids, cloud asset metadata, k8s pod ids, etc. These fields and their values are often important for interpreting the log message correctly. In order to display them, the user can currently add an additional column for each relevant field. This takes up a lot of horizontal screen space, though, leaving less for the actual message. It also leads to a lot of blank space on screen when the resource fields are sparse due to mixed origins of the log entries.

We could improve that by defining a "virtual" column that combines the values of multiple fields and renders them in a space-efficient way. The particular representation would depend on the combination of fields that exist in each log entry document. Examples of such fields could be the following:

• service.name
• container.id
• host.name
• cloud.instance.id

OpenTelemetry also defines a set of metadata fields in its resource semantic conventions to form a resource context.

🎨 Design help needed

Marker column

For some properties of log entries the information that a field is present or indicates a certain circumstance is more important to see at a glance than the full value of the field. If we were to represent these row characteristics as small icons, badges, or similar visual cues, we could efficiently convey them to the user without using the space for a full column. This pattern is established in code editors as a "marker" or "gutter" column to convey boolean or categorical information about a line tersely. Examples if indicators in such a marker column could be:

• "This document was not parsed correctly during ingestion." (i.e. has _ignored fields)
• "This document has a multi-line stack trace attached." (which would be best viewed in the detail fly-out)
• "This document has triggered an alert." (if we were able to annotate the alerts accordingly)

🎨 Design help needed

Content column

The timestamp and message of a log entry usually constitute the most basic and most important pieces of information. But not all documents homogeneously carry that message in the message field. By design or due to ingestion errors some types of entries have message-like content in fields like error.message or event.original instead. Adding separate columns for these would (again) take up a lot of screen estate and make it harder to read other columns. A virtual field could combine these message-like fields or display the most relevant using a heuristic without wasting space.

Additional considerations

Consistency and transparency

By the nature of this feature, a virtual field does not necessarily correspond to a single field in the document directly. Nevertheless, the user should be able to understand which fields the combined representation was derived from. This is important so they can still use these fields in queries, filters, break-downs and other situations in which the combined form is not available. To that end and representation should, at least in their detail view

• indicate the field names and values used
• provide actions to make use of these values (such as adding them as filters)

Usage in the field detail pop-over

In the data grid that underlies the log entry table, each cell can optionally be expanded into a pop-over. These pop-overs offer the chance for us to show a spacious and possibly more comprehensive representation. They also enable us to offer a larger set of actions to the user than the hover icons in the grid cell would allow.

Usage in the log entry detail fly-out

The log entry detail fly-out should afford at least the same level of convenient representation as the grid row. Akin to the cell pop-over it allows for a more spacious representation. In the interest of consistency the formatting code should be shared between both.

Implementation proposal

Inject the columns into the Discover state

The source of truth for the columns displayed in the document table is the column property on the Discovper app state. For every entry (except for _source) the grid will try to render a column, regardless of whether it's part of the search hit or not. The particular rendering is delegated to the renderer as described in the next section. The log explorer already sets the default columns upon initialization, so this should work analogously.

Once the entry is included in the columns array, it's also shown in the field list and can be removed like other columns. But once it has been removed, since it's not part of the data view, it will vanish from the field list. In order to enable the user to add these virtual fields again, the field list would have to be extended with a mechanism to keep these special fields permanently available.

Define an external custom renderer

The unified data grid used in Discover has a default renderer for all cells, which applies field formatter overrides defined in the data view and falls back to field-type-specific formatters otherwise. In addition, the unified data grid has a property externalCustomRenderers using which cell renderer overrides can be defined programmatically for columns of a certain name. It is currently not yet exposed via a customization point in the main Discover component, but once it is the Log Explorer could pass renderers that match the names of the virtual columns injected above.

The renderer is responsible for both the "cell representation", which should be optimized for space efficiency as well as the "popover view", which offers room for more elaborate representations. The two cases are distinguished via the isDetails property.

Define cell actions

The mechanism to influence the cell actions is based on the plugins/ui_actions registry. That means it involves three parts:

• register a new trigger id
• register actions for the new trigger id
• get the document grid to use the actions registered for the new trigger id

To override Discover's hard-coded default cell actions the property cellActionsTriggerId can be passed to the unified data table. Similar to the external custom renderer there would have to be a customization point to pass that to the Discover main container. In order to still offer the built-in actions, these would have to be attached to the new trigger id too. The security ui team's usage can serve as an example of how to achieve that.

Actions can be guarded from being displayed based on the field by implementing the isCompatible function on it.

[gbamparop]

It seems to also be related to https://github.com/elastic/logs-dev/issues/43

[ruflin]

Nice write up at @weltenwort .

keep these special fields permanently available

I would go so far that these fields not only cannot be removed but also not changed in order. Columns can only be added after these virtual columns.

[weltenwort]

I would go so far that these fields not only cannot be removed but also not changed in order. Columns can only be added after these virtual columns.

I can see how that would work for smaller columns like the markers. But for larger columns like the resource context or the "content" it might be problematic to use up a lot of space without the ability for the user to adapt it to their use-case.

[weltenwort]

ℹ️ Since some of the designs move the "markers" to the front or actions to the end, I want to add here that it's also possible to add non-removable "control columns" to the beginning and end. That's a bit simpler, but still requires exposing a customization point on the <DiscoverContainer />.

[weltenwort]

@gbamparop do we want to consider this closed now that you created individual issues? (thank you for that)

[gbamparop]

@gbamparop do we want to consider this closed now that you created individual issues? (thank you for that)

Sounds good, they reference this issue as well in case more information is needed.