[Lens] Heatmap / Swim lane integration #97978

darnautov · 2021-04-22T10:32:12Z

Summary

Adds Heatmap/Swim lane chart by https://github.com/elastic/elastic-charts to the Lens editor.

If you encountered any rendering-related issues please check a list of known issues and create a bug report if necessary.

Known issues/limitations

elastic-charts related

Value formatting doesn't apply to the legend labels. It's been fixed in the elastic-charts, should be part of the upcoming patch release.
Intervals don't work 100% reliably (Heatmap time buckets should use value from timeScale elastic-charts#1165). We made a decision to disable xDomain until the issue is resolved on the elastic charts, so be aware the chart won't auto-extend with respect to the time range.
Legends items might contain duplicated 0 (Heatmap legend with linear color scale contains duplicate 0 elastic-charts#1166)
Legends is missing entries in some cases ([Heatmap] Legend occasionally missing some entires elastic-charts#1191)
Overlapping labels ([Heatmap] Rotate or hide overlapping labels on X-axis elastic-charts#1168)
Overlay mask for brushing over categorical data isn't rendered (Heatmap brushing action for categorical data elastic-charts#1170)
Brushing action over the numeric categorical data crashes the browser's window (Brushing action crashes the the browser's window elastic-charts#1171)
axis labeling (need charts issue and then lens issue)
value labels (needs charts issue? and then lens issue)

require changes on the Lens side

Axis ordering. The heatmap chart supports xSortPredicate and ySortPredicate (at the moment using dataIndex for both). Axis configuration in Lens should allow configuring the sorting function. similar to: covered in this issue and NOT required for removal of Experimental badge: [Lens] Allow client-side sorting of dimensions and legends at datasource level for all chart types #86184
Time range brushing and terms selecting doesn't work at the same time

Heatmap / Swim lane chart behavior

Depending on the data type of the X-axis, the chart looks and behaves slightly differently.

TIme-series data

Chart example:

Brushing action applies a selected time range similar to the XY chart.

Categorical data

Chart example:

The brushing action invokes filters based on selected X and Y values.

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

flash1293 · 2021-05-06T13:19:29Z

List of things I noticed:

Suggestions - when on broken down xy chart a lot of suggestions are rendered. I think we can solve this by this by emitting suggestions for reduced tables, but make them hidden (only showing it for unchanged tables)
As discussed, emit hidden suggestions for tables with just a single bucket dimension (mapping it to x axis and rendering without y axis)
Just in general, render the heatmap if there is no vertical axis as it still makes sense
Do not emit visible suggestions if heatmap is already the current visualization
Legend has two entries for "> 0"

The top legend often looks broken - should we default to right legend?
Clicking a suggestion breaks Lens
Do not generate an expression in toExpression if metric is missing - right now it's just rendering an empty screen, but you can simply return null as an expression in this case
Why not allowing date histograms on the vertical axis? I think it makes sense to default it to the horizontal axis but IMHO we shouldn't completely prevent it
Filters by clicking not working yet

Let me know once the code is ready for a more detailed loo

flash1293 · 2021-05-12T08:50:46Z

I started reviewing again, but it seems like most things from the list above are not addressed yet. Let me know once I can take another look. A few other things I noticed:

If y axis is missing, an error is shown now. This should render with a single "row" instead so the user sees what they configured so far (similar to

kibana/x-pack/plugins/lens/public/xy_visualization/expression.tsx

Line 713 in 6b8c44d

if (!xAccessor) {

)
There are a lot of timezone errors because your are directly using it from uiSettings - however the default setting is Browser which means the actual time zone has to be inferred from the browser. This is how xy chart is doing it:

kibana/x-pack/plugins/lens/public/xy_visualization/index.ts

Line 22 in a9d0b6f

function getTimeZone(uiSettings: IUiSettingsClient) {
Starting from a broken down xy chart, there are two heatmap suggestions (because there are two tables with different nesting of the dimensions):

* I don't think it makes sense to show both, we should probably hide suggestions based on the `reorder` change type

flash1293

I found one other thing - if "Last value" is used as metric, it's possible array values are returned (like for products.price in the ecommerce sample data). In this case the chart is rendering "No results found".

This is fine for me, but on xy chart we show a warning in this case to make the user aware:

Heatmap should do the same thing - it's this piece of code ( I guess it makes sense to :

kibana/x-pack/plugins/lens/public/xy_visualization/visualization.tsx

Line 418 in 51616e1

getWarningMessages(state, frame) {

Otherwise this looks fine to me - going to approve after that.

@wylieconlon I think it's worth taking another look at that from your side if you have the time.

wylieconlon

Yes, I found the same issue as @flash1293 with the "Last value" function. It looks like it's actually a bug because the heatmap fails to render when using last value. You can reproduce this using the kibana_sample_data_ecommerce dataset using any products.* fields.

I also found another bug, the heatmap legend is not formatted. You can test this by using the Value formatter option on the metric, or by adding a default formatter to your index pattern:

Other than these new issues, I think my previously found issues were all fixed.

flash1293 · 2021-06-03T15:24:07Z

It looks like it's actually a bug because the heatmap fails to render when using last value

It looks like right now it's excluding all array values, but rendering single values if there are any. IMHO this is an acceptable behavior for the beta version as long as we have the warning

wylieconlon · 2021-06-03T17:24:07Z

@flash1293 The case I was running into is when every value is an array, the chart is completely blank, instead of showing an empty state or warning message.

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.tsx

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.scss

MichaelMarcialis

Left one additional small comment on re-review. Also responded to my previous comment on the EUI class targeting and styles with a possible solution.

Otherwise, this looks good from my perspective. Will approve once these changes are made. Thanks!

darnautov · 2021-06-04T09:43:10Z

@wylieconlon I've created a PR in the elastic-charts repo to address an issue with the legend items formatting

darnautov · 2021-06-04T09:54:05Z

Thanks for taking another look @MichaelMarcialis, I resolved your comments about the chart switcher in d699e9e

flash1293

LGTM

Left two nits I would like to get addressed, but I don't think I have to review again for them.

Not sure whether we noted that down somewhere, but the legend seems to miss entries sometimes:

No entry for the dark shade of red. Seems to be an elastic-charts issue

flash1293 · 2021-06-04T11:32:30Z

x-pack/plugins/lens/public/heatmap_visualization/chart_component.tsx

+  const valueFormatter = formatFactory(valueColumn.meta.params);
+
+  // Enable xDomain when https://github.com/elastic/elastic-charts/issues/1165 is resolved.
+  // @ts-ignore


Nit: We usually don't keep these thins in code, can you remove it from here and move it into an issue?

Removed in 9bb03df

flash1293 · 2021-06-04T11:37:56Z

x-pack/plugins/lens/public/heatmap_visualization/chart_component.tsx

+
+  // @ts-ignore
+  const onElementClick: ElementClickListener = (e: HeatmapElementEvent[]) => {
+    const cell = e[0][0];


nit: Please cast the (e: HeatmapElementEvent[]) => void to ElementClickListener instead of ignoring the ts error. That's retains at least some type safety.

Fixed in ef95471

wylieconlon

LGTM, this is working very well in all the cases I could test!

I did find a new charts bug in my latest round of testing: elastic/elastic-charts#1192

MichaelMarcialis

Thanks for making those changes. I left two last nitpicky comments, but nothing worth holding you up for. Approving.

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.tsx

* heatmap wip * format values on X axis * format values for cells * show labels * support legend configuration * render preview * add icon * [ML] update visualization * [ML] init suggestions * [ML] fix preview * [ML] fix groupPosition for the legend control * [ML] add formatter for Y-axis * [ML] filterOperations for cell value * [ML] fill all available height * [ML] delete unused file * [ML] fix suggestion state * [ML] update suggestion, add hiding logic * [ML] suggestions unit tests * [ML] rename legend interface * [ML] unit tests for visualization, add error messages * [ML] fix typos in xy visualization tests * [ML] support click event * [ML] add xDomain for time series data * [ML] support empty Y axis * [ML] change legend default position * [ML] getTimeZone util * [ML] hide suggestions for reorder * [ML] support brush event * [ML] update unit tests * [ML] render grid lines in preview * [ML] don't display errors on init * [ML] utilize chartsThemeService * [ML] support histogram for the vertical axis * [ML] fix clearLayer * [ML] show empty placeholder on no data * [ML] fix X domain min * [ML] reject suggestions for 3 or more buckets * [ML] suggestions for histograms and histogram for Y-axis * [ML] fix unit tests * [ML] update suggestions for active heatmap * [ML] chart data test for heatmap * [ML] test for transitioning from heatmap to barchart * [ML] disable xDomain * [ML] support intervals in axes configurations * [ML] hide label on the vertical axis when there is only a horizontal dimension * [ML] set min cell height for better suggestions preview rendering * [ML] fix tooltip for empty vertical axis config * [ML] fix click and brushing for empty Y axis * [ML] update functional test * [ML] show beta label * [ML] fix legend control * [ML] dataIndex sort by default for the X axis * [ML] use euiPaletteForTemperature with quantize color scale * [ML] hide all suggestions * [ML] fix chart data extension issue * [ML] fix the caret symbol positioning * [ML] update unit tests for the heatmap suggestions * [ML] replace EuiBetaBadge with EuiBadge * [ML] update functional test * [ML] fix chart switch styles * [ML] fix functional test * [ML] return null instead of expression with a missing value accessor * [ML] use table id as a chart id * [ML] fix scale type for a single row of data * [ML] filter out undefined values * [ML] fix isXAxisLabelVisible * [ML] update chart_switch styles * show warning message for the array values * remove unused code * replace ts-ignore with manual type casting * add unit tests for error and warning messages * add css class for append, conditional flex group

kibanamachine · 2021-06-04T19:31:45Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* heatmap wip * format values on X axis * format values for cells * show labels * support legend configuration * render preview * add icon * [ML] update visualization * [ML] init suggestions * [ML] fix preview * [ML] fix groupPosition for the legend control * [ML] add formatter for Y-axis * [ML] filterOperations for cell value * [ML] fill all available height * [ML] delete unused file * [ML] fix suggestion state * [ML] update suggestion, add hiding logic * [ML] suggestions unit tests * [ML] rename legend interface * [ML] unit tests for visualization, add error messages * [ML] fix typos in xy visualization tests * [ML] support click event * [ML] add xDomain for time series data * [ML] support empty Y axis * [ML] change legend default position * [ML] getTimeZone util * [ML] hide suggestions for reorder * [ML] support brush event * [ML] update unit tests * [ML] render grid lines in preview * [ML] don't display errors on init * [ML] utilize chartsThemeService * [ML] support histogram for the vertical axis * [ML] fix clearLayer * [ML] show empty placeholder on no data * [ML] fix X domain min * [ML] reject suggestions for 3 or more buckets * [ML] suggestions for histograms and histogram for Y-axis * [ML] fix unit tests * [ML] update suggestions for active heatmap * [ML] chart data test for heatmap * [ML] test for transitioning from heatmap to barchart * [ML] disable xDomain * [ML] support intervals in axes configurations * [ML] hide label on the vertical axis when there is only a horizontal dimension * [ML] set min cell height for better suggestions preview rendering * [ML] fix tooltip for empty vertical axis config * [ML] fix click and brushing for empty Y axis * [ML] update functional test * [ML] show beta label * [ML] fix legend control * [ML] dataIndex sort by default for the X axis * [ML] use euiPaletteForTemperature with quantize color scale * [ML] hide all suggestions * [ML] fix chart data extension issue * [ML] fix the caret symbol positioning * [ML] update unit tests for the heatmap suggestions * [ML] replace EuiBetaBadge with EuiBadge * [ML] update functional test * [ML] fix chart switch styles * [ML] fix functional test * [ML] return null instead of expression with a missing value accessor * [ML] use table id as a chart id * [ML] fix scale type for a single row of data * [ML] filter out undefined values * [ML] fix isXAxisLabelVisible * [ML] update chart_switch styles * show warning message for the array values * remove unused code * replace ts-ignore with manual type casting * add unit tests for error and warning messages * add css class for append, conditional flex group Co-authored-by: Dima Arnautov <[email protected]>

kibanamachine · 2021-06-09T19:12:03Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`lens`	651	670	+19

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	1.3MB	1.3MB	+45.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`lens`	34.6KB	38.1KB	+3.4KB

History

💚 Build #129269 succeeded 6d35fb7
💚 Build #129166 succeeded 5a25c36
💚 Build #129139 succeeded 4cb2f40
💚 Build #129129 succeeded d699e9e
💚 Build #128883 succeeded 3cb6b14

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @darnautov

darnautov added 18 commits April 6, 2021 09:23

heatmap wip

d67fad6

format values on X axis

381240e

format values for cells

415b567

show labels

9fbca6a

support legend configuration

c25cf3e

render preview

2c65a8d

add icon

54e96a0

Merge remote-tracking branch 'upstream/master' into lens-heatmap-chart

38a5a6d

[ML] update visualization

cd912c2

[ML] init suggestions

c5d02d0

Merge remote-tracking branch 'upstream/master' into lens-heatmap-chart

2b40fe3

[ML] fix preview

7aaeccd

[ML] fix groupPosition for the legend control

bb4910c

[ML] add formatter for Y-axis

48bbad0

[ML] filterOperations for cell value

398b92f

[ML] fill all available height

d25f7e9

[ML] delete unused file

1e6d858

[ML] fix suggestion state

a8b3be4

Merge remote-tracking branch 'upstream/master' into lens-heatmap-chart

3d89c72

darnautov self-assigned this May 11, 2021

darnautov added 5 commits May 11, 2021 12:29

[ML] update suggestion, add hiding logic

62db0f2

[ML] suggestions unit tests

d7b6d01

[ML] rename legend interface

c252224

[ML] unit tests for visualization, add error messages

5583f3b

[ML] fix typos in xy visualization tests

b8ac2d3

darnautov added 3 commits May 12, 2021 11:29

[ML] support click event

ab10566

[ML] add xDomain for time series data

1b9d09f

[ML] support empty Y axis

f7b53ce

flash1293 reviewed Jun 3, 2021

View reviewed changes

wylieconlon reviewed Jun 3, 2021

View reviewed changes

MichaelMarcialis reviewed Jun 3, 2021

View reviewed changes

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.tsx Outdated Show resolved Hide resolved

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.scss Outdated Show resolved Hide resolved

MichaelMarcialis reviewed Jun 3, 2021

View reviewed changes

[ML] update chart_switch styles

d699e9e

darnautov requested a review from MichaelMarcialis June 4, 2021 09:51

show warning message for the array values

4cb2f40

flash1293 approved these changes Jun 4, 2021

View reviewed changes

darnautov added 3 commits June 4, 2021 14:32

remove unused code

9bb03df

replace ts-ignore with manual type casting

ef95471

add unit tests for error and warning messages

5a25c36

darnautov requested a review from wylieconlon June 4, 2021 13:11

wylieconlon approved these changes Jun 4, 2021

View reviewed changes

MichaelMarcialis approved these changes Jun 4, 2021

View reviewed changes

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.tsx Outdated Show resolved Hide resolved

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/chart_switch.tsx Outdated Show resolved Hide resolved

add css class for append, conditional flex group

6d35fb7

darnautov enabled auto-merge (squash) June 4, 2021 17:45

darnautov added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 4, 2021

darnautov merged commit 0793753 into elastic:master Jun 4, 2021

kibanamachine mentioned this pull request Jun 4, 2021

[7.x] [Lens] Heatmap / Swim lane integration (#97978) #101426

Merged

darnautov deleted the lens-heatmap-chart branch June 4, 2021 21:27

ghudgins mentioned this pull request Jun 9, 2021

[Lens] mark heatmaps "experimental" instead of "beta" #101815

Closed

peteharverson mentioned this pull request Jul 5, 2022

Embed Swimlane Visualization in Kibana #23492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Heatmap / Swim lane integration #97978

[Lens] Heatmap / Swim lane integration #97978

darnautov commented Apr 22, 2021 •

edited by ghudgins

Loading

flash1293 commented May 6, 2021 •

edited

Loading

flash1293 commented May 12, 2021

flash1293 left a comment

wylieconlon left a comment

flash1293 commented Jun 3, 2021

wylieconlon commented Jun 3, 2021

MichaelMarcialis left a comment

darnautov commented Jun 4, 2021

darnautov commented Jun 4, 2021

flash1293 left a comment

flash1293 Jun 4, 2021

darnautov Jun 4, 2021

flash1293 Jun 4, 2021

darnautov Jun 4, 2021

wylieconlon left a comment

MichaelMarcialis left a comment

kibanamachine commented Jun 4, 2021

kibanamachine commented Jun 9, 2021

[Lens] Heatmap / Swim lane integration #97978

[Lens] Heatmap / Swim lane integration #97978

Conversation

darnautov commented Apr 22, 2021 • edited by ghudgins Loading

Summary

Known issues/limitations

elastic-charts related

require changes on the Lens side

Heatmap / Swim lane chart behavior

TIme-series data

Categorical data

Checklist

flash1293 commented May 6, 2021 • edited Loading

flash1293 commented May 12, 2021

flash1293 left a comment

Choose a reason for hiding this comment

wylieconlon left a comment

Choose a reason for hiding this comment

flash1293 commented Jun 3, 2021

wylieconlon commented Jun 3, 2021

MichaelMarcialis left a comment

Choose a reason for hiding this comment

darnautov commented Jun 4, 2021

darnautov commented Jun 4, 2021

flash1293 left a comment

Choose a reason for hiding this comment

flash1293 Jun 4, 2021

Choose a reason for hiding this comment

darnautov Jun 4, 2021

Choose a reason for hiding this comment

flash1293 Jun 4, 2021

Choose a reason for hiding this comment

darnautov Jun 4, 2021

Choose a reason for hiding this comment

wylieconlon left a comment

Choose a reason for hiding this comment

MichaelMarcialis left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 4, 2021

💚 Backport successful

kibanamachine commented Jun 9, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

darnautov commented Apr 22, 2021 •

edited by ghudgins

Loading

flash1293 commented May 6, 2021 •

edited

Loading