[Canvas][meta] Unify Expression Renderers and Deprecate flot

[clintandrewhall]

Background

At its inception, Canvas created a novel means of data manipulation and rendering we now call expressions, where a set of string-based "steps" take input and supply output to the next step. Expressions as both a concept and runtime were then extracted out into the expressions plugin in Kibana. To date, more products such as Lens use Expressions to manipulate and visualize data.

While the Expressions tooling was successfully extracted, the rendering portion of the Expressions-- functions such as revealImage, plot, shape, etc-- were tightly coupled to Canvas and remained untouched. Many of these renderers utilize an older jQuery plugin known as flot to render charts.

In the time since these renderers were written, Elastic Charts has become the preferred (and frankly, better) method of data visualization, while flot has fallen out of favor and into a barely-maintained library. With each passing release, the Canvas chart renderers become more and more difficult to maintain, and thus have been slated for replacement with Elastic Charts for quite some time.

While one might think the biggest bottleneck migrating these renderers from flot to Elastic Charts might be time, in reality, it's release strategy: changing the renderers to use a different charting library constitutes a breaking change. As a "pixel perfect" presentation solution, users should not upgrade to the next minor and have their Canvas workpads visually change, or even break.

We originally planned these changes for 8.0. After spending some time researching these updates as breaking changes, a better strategy emerged: unification and deprecation.

Points of Contact

• Canvas: @crob611
• Lens: @flash1293
• App Services: @ppisljar
• EPAM: @Kunzetsov

Reference

• 8.0 Research: [Canvas] Remove Flot from Canvas Visualizations #83029
• Alignment proposal: Cross App Renderer Alignment #95902

Considerations

• There are currently several expressions-backed plugins in Kibana. They exist in src/plugins and begin with the vis_type_ prefix.
  • These plugins were written for specific solutions, like Lens. They are not necessarily designed for re-use.
  • @ppisljar has written about resolving and refactoring these plugins in Cross App Renderer Alignment #95902
• We'll either be creating new plugins for functions that currently exist in Canvas but not elsewhere-- revealImage, shape, etc-- or working with other teams to refactor existing plugins-- visTypeXY, visTypeMarkdown, etc-- to meet everyone's needs.
• Not all rendering functions need to be extracted. Functions that are unique to the Canvas context-- filters, embeddable, etc-- should not be extracted.

Implementation Considerations

• Keep theming in mind for these renderers.
• Use Emotion "CSS-in-JS" to style the output.
  • Consult with @cqliu1 regarding status in Canvas, and align with that project.
• Add telemetry and tests as we go!

Project Plan

This list is not exhaustive, nor linear-- some of these tasks can be completed in parallel.

• Write a guide for creating new visualization plugins based on the vis_type_* plugins that currently exist.
  • Evaluate structure for improvements.
  • Consider the Canvas lifecycle, extensibility, etc.
  • The end goal should be to guide anyone, within Elastic or beyond, in creating a visualization that will "just work".
• Using the guide, create new visualizations based on Canvas rendering functions that currently exist, revising the guide as necessary:
  • revealImage
  • shape
  • error
  • repeatImage
  • image
  • metric
  • progress
• Devise a strategy to deprecate the existing elements and functions in Canvas with the new visualizations:
  • Create UI supporting deprecated elements in older workpads, opting in for newer workpads.
  • Create method of supporting deprecated functions in the expressions editor.
  • etc.
• With each completion, deprecate and integrate new visualization. Consider using Labs to gate the changes as we go.
• Work with Expressions and Lens to align existing vis_type_ plugin APIs for use in Canvas.
• Refactor existing visualizations, and repeat above.

Long Term

• Track telemetry and migrate deprecated renderers to optional, external plugin or package.
  • Delete from Canvas.

Result

• From this work, particularly if we use Labs, we'll be able to deprecate and replace visualizations on a rolling basis through 7.x and into 8.0.
• Canvas will gain new visualizations that already exist, like vis_type_timelion and vis_type_tagcloud.
• Other solutions will gain new visualizations that have only existed in Canvas, like gauge and metric.
• Canvas bundle size will drastically decrease, and we'll gain performance and consistency.

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/datavis (Team:DataVis)

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[ppisljar]

i can try working on an initial version of that vis_type guide. one thing to note is that we probably don't want to create actual vis types (something that would already appear in visualize editor) but just the expression function and renderer part. the main thing we'll need to address (besides moving some files) is described in this issue #74644 (and should also be done for all renderers that are not extracted out of canvas)

[clintandrewhall]

@ppisljar you'll want to touch base with @crob611, as he's already started a guide, (it's on the Canvas project plan). There's no doubt your input will be needed... where would you think the markdown should live? Expressions plugin?

[crob611]

@ppisljar @clintandrewhall Do you think we should use the vis_type_X naming scheme for those that, like you mention, aren't actually going to appear in the visualization editor? Any idea on a better naming scheme? Maybe like expression_X?

And yes @ppisljar I'm putting together a guide. Should be wrapping that up shortly.

[ppisljar]

i am ok with using vis_type_X still, but i am also ok with expressions_X.

i think the markdown (lets make sure its .mdx) should go under dev_docs (cc @stacey-gammon do you agree?) as its gonna be describing the good practices to creating expression functions and renderers. we already have a very simple tutorial there on how to register a function and a renderer for expressions. we could either extend those sections or (imo preferably) link from there to this more extended guides on writing fn/ renderer and the best practices around it.

[clintandrewhall]

Perhaps we go with expressions_x for all the new ones, and rather than refactor vis_type_foo in place, let's write a new expressions_foo that people can migrate to...?

[ppisljar]

sounds good to me. the only thing i am not sure is if we want to refactor vis_type_foo to expression_foo at all, as the vis_type plugins will probably contain their vis part for quite some time (until we get rid of visualize at least) and i don't think spliting this in two makes much sense (expression_foo + vis_type_foo using expression_foo under the hood) but would be possible.

[clintandrewhall]

Gotcha... maybe just rename the plugin from vis_type_foo to expression_foo when the refactoring is complete.

[crob611]

👍 We'll go with expression_foo for the functions/renderers that we pull out of Canvas

[clintandrewhall]

@ppisljar, @corey has started revisions to the guide here: #101611

[ghudgins]

just checking in on this, for 8.0, do you need anything from @elastic/datavis on this?

[ThomThomson]

Closing out this initiative for now as it has been mostly completed, and we are unable to prioritize the remaining portions.