Visualization Library Redesign

[stormpython]

Currently the visualization library needs to be redesigned.

Problems

1. Vislib is too tied to Kibana
   • This makes prototyping new charts/visualizations difficult and slow since many different requirements are needed before a visualization can even be rendered, such as data in the correct format
   • Changing or expanding options for charts requires changes at the chart level instead of passing changes into a config || {} object.
2. The way the vislib works is difficult to understand and not a standard approach to how most d3 reusable charts libraries are created.
   • Since we work on an open source project, the readability of the code should be such that anyone with the appropriate skill set can contribute without getting lost in how the pieces are tied together.
   • If a developer from the community wanted to contribute a visualization type to Kibana, it would require a massive undertaking currently.
   • While we have components, e.g. x-axis, none of our charts actually stand alone. Rendering a line chart doesn't actually render a line chart, it simply renders lines. While having components is good for combining components into complex charts, I believe we need modular charts that work on their own. Everything does not have to be componentized, especially one off charts like bar, line, or area.
3. It is difficult to understand what chart options are actually available given that there is no real list of API features.
   • This is due in part to how the vislib is written.

Proposed Solutions

1. Move the Vislib out of the Kibana repo
   • The Vislib should stand alone and be re-usable across any number of projects.
   • This would improve the speed of prototyping new charts, allow for more defined chart APIs, improve the ease of testing visualizations, and separate concerns better.
2. Create charts in the form of Towards Reusable Charts by Mike Bostock
   • This will allow D3 plugins to be easily used within the Vislib which means developers can easily add new visualizations to the library without having to do much work.
   • It is also a standard way of creating reusable charts, so other developers well versed in D3 will have a good understanding of what is going on in the code.
   • It makes it easy to spin up new charts and have well documented APIs for those charts.
   • It allows for a considerable amount of flexibility when it comes to adding options to charts.
3. Change the design of the Vislib
   • This is by far the most controversial item of the proposals.
   • It is difficult to understand how the Vislib works without a thorough code review and adding new features or charts takes a lot longer since many pieces are broken up across the library or not broken out enough.

Vislib Design Proposals

Current Design

var newChart = new Vislib($elem, options);
newChart.render(data);

Angular Directive Design

<visualize-chart ng-repeat="chartData in data">
  <line-chart data="chartData" opts="options"></line-chart>
  <line-chart data="chartData" opts="options"></line-chart>
  <line-chart data="chartData" opts="options"></line-chart>
</visualize-chart>

ReactJS Design

<BarChart data={barData} options={...}/>

Object Oriented Designs

var newChart = new Vislib($elem).data(data, options).render();

var newChart = new Vislib($elem, data, options).render();

var newChart = new Vislib({
  el: $elem,
  data: [....],
  options: {...}
}).render();

// Many more proposals of this type

**It was also floated that we drop D3.js entirely for a standard charting library or something that utilizes canvas.

Implementation Issues

A lot was discussed regarding how we implement the visualization library. For example, one of the future requirements will be to print visualizations as pdfs. So the idea was floated that all visualizations, including those that have multiple charts be placed in 1 SVG.

There are several problems with the 1 SVG approach.
* Placing SVGs within SVGs doesn't work that well, so all chart elements will need to be placed within g elements.
* Placing everything in g elements could potentially hurt performance.
* Placing everything in g elements means that in order to make sure everything is positioned correctly we will need to transform everything within a g element instead of relying on html divs and flexbox for proper spacing and flexing. This adds to the complexity of the code and introduces many places where things could fall apart when you have lots of different components that need to be transformed.

---

Another issue that is related to the one above is the layout. Currently, the layout is specified by the vislib. So, before any chart can be prototyped, a layout needs to be provided in JSON format. So if any new developer wants to contribute to the vislib, they will have to specify a specific layout for their visualization. This makes it really hard to change how things are laid out. For example, moving a legend from the right side to the left side requires a new layout. Another issue has to do with the fact that rotating labels on the x-axis causes a complete redraw so that both the y-axis and the x-axis can be of the appropriate size to display all the relevant values. A proposed work around for the x-axis/y-axis resizing issue was to use cassowary, which is a constraint solver to deal with resizing issues between the two axes.

---

Whenever anything changes in a visualization, the entire object is destroyed and has to be rebuilt again. This is not ideal. Part of this has to do with the layout having to be rebuilt if anything flexes and the fact that we don't expose an API for redrawing the visualization without having to rebind the data, but simply pass in a new width and height to the vislib object.

---

It feels many times as if the vislib doesn't really work with the way the application works. One example being issue #3359. Here we have an issue where the timing of the vislib and the Angular DOM is off, causing us to have to re-render until the DOM is ready.

---

Note: This is by far not a comprehensive list of the issues facing the vislib or all the possible solutions discussed. Its simply a summary of my thoughts and what I gleaned from our discussion last week. @rashidkpc , @spalger, @w33ble, @simianhacker, @panda01, @lukasolson, @jthomassie feel free to add to this ticket any glaring wholes or gaps I might have missed.

[gthorsen]

This would be very helpful, as currently I have a business need for a Sankey chart, but no great way to easily plug that into Kibana.

[ajrasch]

This would be OUTSTANDING!

[chenryn]

+1.
@gthorsen Have you implemented the sankey visualize for kibana now?

[yehosef]

+1 I like the direction

[maheshnayak1]

+1

[ppisljar]

I agree,

1. we should take vislib out of kibana
2. it should be easy to add new visualizations independent from kibana

you mentioned there was some discussion about using standard charting library. where did that end ? i think there are quite some good open source charting libraries out there and it might be better than starting from 0 ?

I propose that we split chart settings into several categories (accordion style ?). First step remains the same where user selects one of the chart types (like area, bars, ...). All of the below only applies to point series chart types (area, bar, line) and does not apply to charts like pie or map tiles. Then user needs to setup his data (query, aggregations and metrics). After he's done he can already preview his default chart. Then he would click on the options where he could configure chart in details (based on the selected data).

For each point serie he could configure how its charted (so we can have a combination of line and bars for example) #4566

• general
  • orientation: Horizontal bar chart #1396 horizontal (x axis for categories) or vertical (y axis for categories)
  • allow export: Export chart to image #1366 Export to PDF  #509 enable/disable export functionality
• appearance
  • colors
  • font, font size Configurable font size #2207
  • depth (3D visualizations #2729)
  • background
  • borders,
  • margins
• baloons
  • formating ( Tooltip aggs #3090 ? )
  • position
  • appearance
• legend
  • enabled: allows enabling/disabling legend
  • position: Legend positioning: top/left/bottom/right/none #572 allows setting legend to top/bottom/left/right
  • appearance: allows setting background color, border, color, margin, padding)
• cursor
  • enabled: allows enabling/disabling cursor
  • appearance: line color, type of line
• category axis:
  • position: top/bottom for horizontal axis or left/right for vertical
  • formating: Per visualization field formatters #4957
  • appearance:
• titles array (allows adding multiple titles)
  • position: title position (top, bottom, left, right)
  • appearance: font size, color, margin
• value axes Second Y-Axis #2390 (allows adding multiple value axes)
  • position: left/right or top/bottom
  • title: title of axis
  • type: normal, log Logarithmic scale for bar charts #5708
  • stacking type: none, normal (values are stacked), 100% (100% stacked), 3d 3D visualizations #2729
  • appearance: color, margins
• graphs Multiple chart types per plot #4566 (allows adding multiple graphs),
  • data (which serie is belonging to this chart)
  • type: line, area, bars, smoothed line,
  • value axis: which value axis is to be used with this graph
  • appearance: color, fill color, markers
• trend lines Trend lines on charts #3325 Add Threshold Lines in Charts #5037 (allows adding multiple trend lines)
  • type:
  • value axis: which value axis is to be used with this trend line
  • appearance: color

in the code we would separate chart simillary (graphs, category axis, value axes, trend lines...). Whole chart should be configured with a config object (all the settings from above) and data object (results from elasticsearch query).

[ppisljar]

I was looking at what is available at the moment in terms of existing charting libraries. Not that we should go for one of them but just to better understand the problem. I am going to post few links and some taughts. Please let me know if that's not appropriate.

here is a nice website which keeps quite up to date with charting libraries available.

on the note mentioned by @stormpython that we should drop D3 for a library using canvas, it seems that the majority of libraries out there are using SVG to render the charts, the same as D3 is doing. That goes thru for both commercial and open source products.

out of open source products i found out this two to stand out:

NVD3 is build on top of D3 with a goal to make reusable charts and not take away any power of D3. Charts look quite nice (could use some love) and there seem to be plenty of options. The library seems to be well maintained.

Polty.JS is also build on top of D3, also tons of options, also looks nice and also seems to be well maintained. On the first look I like this one more.

Next to the opensource products I also found quite some interesting commercial products, if someone is interested to see what they offer:

zingcharts, highcharts, amcharts and fusioncharts are one of the best in my opinion.

[nreese]

dc.js is a neat client side charting package built on the foundation of exploring multi-dimensional datasets. It includes lots of chart types that are designed to be reactive to changes in other charts. The only flaw is that the underlying data engine, crossfilter, requires all data to be loaded into the client. It would be great if dc.js could just sit on top of elasticsearch's rich aggregation framework. Maybe dc.js would be a good place to start for migrating kibana's visualizations outside of kibana.

[ppisljar]

#9044 addressed most of the issues and we'll track the rest separately