Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs Explorer] Add support for virtual columns in the data table #171844

Closed
gbamparop opened this issue Nov 23, 2023 · 2 comments · Fixed by #173732
Closed

[Logs Explorer] Add support for virtual columns in the data table #171844

gbamparop opened this issue Nov 23, 2023 · 2 comments · Fixed by #173732
Assignees
Labels
Feature:LogsExplorer Logs Explorer feature Team:obs-ux-logs Observability Logs User Experience Team

Comments

@gbamparop
Copy link
Contributor

gbamparop commented Nov 23, 2023

📓 Summary

Virtual columns will be introduced in Logs Explorer whose values are created on the fly and are not part of the mapping, aiming to provide useful information to users which is easier to consume.

As part of this work, Discover will be updated to support the virtual columns in the data table.

Related to #169874.

✔️ Acceptance criteria

  • Should update the Discover state when the virtual columns are added
  • Should enable consumers to add icons and tooltips on the column name
  • Should enable consumers to add a custom React component in the cell value

💡 Implementation hints

Inject the columns into the Discover state

The source of truth for the columns displayed in the document table is the column property on the Discovper app state. For every entry (except for _source) the grid will try to render a column, regardless of whether it's part of the search hit or not. The particular rendering is delegated to the renderer as described in the next section. The log explorer already sets the default columns upon initialization, so this should work analogously.

Define an external custom renderer

The unified data grid used in Discover has a default renderer for all cells, which applies field formatter overrides defined in the data view and falls back to field-type-specific formatters otherwise. In addition, the unified data grid has a property externalCustomRenderers using which cell renderer overrides can be defined programmatically for columns of a certain name. It is currently not yet exposed via a customization point in the main Discover component, but once it is the Log Explorer could pass renderers that match the names of the virtual columns injected above.

The renderer is responsible for both the "cell representation", which should be optimized for space efficiency as well as the "popover view", which offers room for more elaborate representations. The two cases are distinguished via the isDetails property.

@gbamparop gbamparop added Team:obs-ux-logs Observability Logs User Experience Team Feature:LogsExplorer Logs Explorer feature labels Nov 23, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

@achyutjhunjhunwala
Copy link
Contributor

Example of Custom Renderer used by the Cloud Security Team-

const customCellRenderer = (rows: DataTableRecord[]) => ({
'result.evaluation': ({ rowIndex }: EuiDataGridCellValueElementProps) => (
<CspFindingRenderer row={rows[rowIndex]}>
{({ finding }) => <CspEvaluationBadge type={finding.result.evaluation} />}
</CspFindingRenderer>
),
'@timestamp': ({ rowIndex }: EuiDataGridCellValueElementProps) => (
<CspFindingRenderer row={rows[rowIndex]}>
{({ finding }) => <TimestampTableCell timestamp={finding['@timestamp']} />}
</CspFindingRenderer>
),
});

@achyutjhunjhunwala achyutjhunjhunwala self-assigned this Dec 13, 2023
achyutjhunjhunwala added a commit that referenced this issue Jan 11, 2024
… virtual column (#173732)

## Summary

- Closes #171844
- Closes #171726
- Closes  #173825

This PR does 4 things

- Customisation point to inject customCellRenderers in the Data Table.
- Adding the 1st virtual column - Content and removal of Message column
- Add cell actions for virtual columns so that the custom fields can be
filtered for
- Renders similar content on the Flyout instead of Message

### To make this PR easy to review

#### Why Kibana Operations

- This PR initially increased the bundle size causing the values to go
beyond the boundaries set in `limits.yml`. This has been fixed now after
suggestion from Kibana Ops team

#### For Kibana Data Discovery Team - Changes done on Unified Data Table

- Moving around types to make them more generic and reusable
- Split the 2 components SourceDocument and SourceDocumentPopover out of
the 1 single large file `ger_render_cell_value.tsx` they were in. This
allows me to export them and lazy load the components in my plugin.
- Passed additional parameters to `externalCustomRenderers` function

#### For Kibana Data Discovery Team - Changes done on Discover Plugin

- Currently the discover plugin was not passing the registered
customisation for `externalCustomRenderers`. Simply added the hook to
retrieve the registered customisation and pass it as props.
- Added new customisation type for `CellRendererCustomisation`

#### For Logs UX Team - Changes done on Log Explorer

- Registered a new customisation for custom Virtual columns.
- Rendering logic for Log Level and Message with fallbacks
- When Message and its fallbacks are not present, we must render the
whole document. Used extracted logic to display complete document as it
appears in current Discover.
- When a JSON column is expanded (expandCellAction), it should render a
JSON viewer as it currently does in Discover
- Items like Log Level which are rendered inside a Chip should allow
contextual actions like Filter In, Filter Out and Copy

## Want to run this PR locally

```
// Checkout this PR
gh pr checkout 173732

// Start Kibana locally
yarn start 

// Start ES locally
yarn es snapshot

// Generate dummy data
node scripts/synthtrace simple_logs.ts --clean

```

## Live Demo can be seen
[here](https://achyutjhunjhunwala-d-pr173732.kb.us-west2.gcp.elastic-cloud.com/app/observability-log-explorer/?pageState=(breakdownField:log.level,columns:!((field:service.name,width:240),(field:host.name,width:320),(field:content)),controls:(data_stream.namespace:(mode:include,selection:(selectedOptions:!(),type:options))),datasetSelection:(selectionType:all),filters:!(),query:(language:kuery,query:%27%27),refreshInterval:(pause:!t,value:5000),rowHeight:0,rowsPerPage:100,time:(from:%272024-01-05T14:39:41.024Z%27,to:%272024-01-05T14:54:43.378Z%27),v:1))

## Demo

![Virtual Column -
Content](https://github.com/elastic/kibana/assets/7416358/db5977cd-1342-4dce-bfe9-bad3ee42487b)

## What's pending - Basically why is the PR still in Draft Stage

- [x] Fix colors for logs levels after confirmation from Isa and hide
level when not present
- [x] Fix JSON rendering on the content cell when message field is not
present so that it looks like exactly how it looks in Discover
- [x] Add E2E tests
- [x] Update DEMO with a gif rather than a screenshot

---------

Co-authored-by: kibanamachine <[email protected]>
delanni pushed a commit to delanni/kibana that referenced this issue Jan 11, 2024
… virtual column (elastic#173732)

## Summary

- Closes elastic#171844
- Closes elastic#171726
- Closes  elastic#173825

This PR does 4 things

- Customisation point to inject customCellRenderers in the Data Table.
- Adding the 1st virtual column - Content and removal of Message column
- Add cell actions for virtual columns so that the custom fields can be
filtered for
- Renders similar content on the Flyout instead of Message

### To make this PR easy to review

#### Why Kibana Operations

- This PR initially increased the bundle size causing the values to go
beyond the boundaries set in `limits.yml`. This has been fixed now after
suggestion from Kibana Ops team

#### For Kibana Data Discovery Team - Changes done on Unified Data Table

- Moving around types to make them more generic and reusable
- Split the 2 components SourceDocument and SourceDocumentPopover out of
the 1 single large file `ger_render_cell_value.tsx` they were in. This
allows me to export them and lazy load the components in my plugin.
- Passed additional parameters to `externalCustomRenderers` function

#### For Kibana Data Discovery Team - Changes done on Discover Plugin

- Currently the discover plugin was not passing the registered
customisation for `externalCustomRenderers`. Simply added the hook to
retrieve the registered customisation and pass it as props.
- Added new customisation type for `CellRendererCustomisation`

#### For Logs UX Team - Changes done on Log Explorer

- Registered a new customisation for custom Virtual columns.
- Rendering logic for Log Level and Message with fallbacks
- When Message and its fallbacks are not present, we must render the
whole document. Used extracted logic to display complete document as it
appears in current Discover.
- When a JSON column is expanded (expandCellAction), it should render a
JSON viewer as it currently does in Discover
- Items like Log Level which are rendered inside a Chip should allow
contextual actions like Filter In, Filter Out and Copy

## Want to run this PR locally

```
// Checkout this PR
gh pr checkout 173732

// Start Kibana locally
yarn start 

// Start ES locally
yarn es snapshot

// Generate dummy data
node scripts/synthtrace simple_logs.ts --clean

```

## Live Demo can be seen
[here](https://achyutjhunjhunwala-d-pr173732.kb.us-west2.gcp.elastic-cloud.com/app/observability-log-explorer/?pageState=(breakdownField:log.level,columns:!((field:service.name,width:240),(field:host.name,width:320),(field:content)),controls:(data_stream.namespace:(mode:include,selection:(selectedOptions:!(),type:options))),datasetSelection:(selectionType:all),filters:!(),query:(language:kuery,query:%27%27),refreshInterval:(pause:!t,value:5000),rowHeight:0,rowsPerPage:100,time:(from:%272024-01-05T14:39:41.024Z%27,to:%272024-01-05T14:54:43.378Z%27),v:1))

## Demo

![Virtual Column -
Content](https://github.com/elastic/kibana/assets/7416358/db5977cd-1342-4dce-bfe9-bad3ee42487b)

## What's pending - Basically why is the PR still in Draft Stage

- [x] Fix colors for logs levels after confirmation from Isa and hide
level when not present
- [x] Fix JSON rendering on the content cell when message field is not
present so that it looks like exactly how it looks in Discover
- [x] Add E2E tests
- [x] Update DEMO with a gif rather than a screenshot

---------

Co-authored-by: kibanamachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:LogsExplorer Logs Explorer feature Team:obs-ux-logs Observability Logs User Experience Team
Projects
None yet
3 participants