Time Series Metric Visualizations

[simianhacker]

This PR is for adding a new time series metrics editor and visualizations to Kibana. It supports the current set of pipeline aggregations and is gear towards time series data sets. It comes with 5 visualization types: Histogram, Gauge, Metric, Top N, and Markdown (with variable substitution).

Feature list:

• Full pipeline aggregation support (including sibling aggs)
• Multiple Y-Axes
• Fully customization histograms (line, area, bar)
• Circle Gauges and Half Circle Gauges with customizable styles
• Percentile bands
• Standard deviation bands
• Time shifting
• Custom index patterns for each series
• Customizable background colors
• Rule based background and text colors for metrics visualization
• Rule based gauge and text colors
• Rule based bars for Top N visualization
• Custom CSS for Markdown visualization
• Markdown variable substitution with live aggregation data
• Annotations for time series histograms with multiple sources
• Ratios based on query strings

[tbragin]

@simianhacker When i tried to build on top of master from last night, I got the following optimization errors. Have you seen that? https://gist.github.com/tbragin/572ddac9e18340376e50e02e4e536621

[simianhacker]

Oops... forgot to add the flot-charts dependencies

[tbragin]

@simianhacker I got it built after the latest update, looking at it now :)

[tbragin]

One issue we must resolve right off the bat is one of naming. Right now in Visualize, all of the options aside from Timelion and this new Timeseries Metric vis are specific chart types. This is what the naming looks like in this WIP PR.

My proposal is to:

1. Group both with the prefix "Timeseries" and differentiate between Metrics and Timelion by naming them "Timeseries: Something" and "Timeseries: Something Else". Alternatively, we can drop "Timeseries" from both, but I sort of like being explicit about the fact that both work with time-series data specifically. (Note: It appears that "time series" is actually usually written as two words, though we currently spell it as one word in the case of Timelion.)

2. We also probably need new icons for these two options.

3. Finally, it may be worthwhile to consider renaming "Metric" to something like "Single-value metric" or "Singlestat" to avoid confusion with the term "Metrics" what I think is central to the goal of thist time-series visualization.

Proposal 1. Discussed with @simianhacker originally.

Proposal 2. The more I thought about it, using "Metrics" for this is ambiguous, as Timelion deals with metric data as well. The main difference between the Timelion approach vs this enhancement is expressions vs visual builder.

@thomasneirynck @ppisljar @uboness @cjcenizal @alt74 @simianhacker @kimchy Thoughts?

[tbragin]

@simianhacker Does this visual builder support doing series ratio? So for instance ratio of errors to overall transactions, the equivalent of the following logic in Timelion: .es(q="error").divide(.es(q="*"))

[simianhacker]

@tbragin For a series ratio's you are limited to metric and pipeline aggregations for the series; sibling aggregations are not available from the bucket_script; this is a limitation set by Elasticsearch. Timelion isn't bound by this limitation because only the basic aggregations are handle in Elasticsearch; everything else is calculated on the Kibana server.

[tbragin]

@uboness and I chatted and discussed the naming - a couple of ideas there:

• "Metric" -> "Number"
• "Timeseries: Visual builder" (will need some other icon, didn't dig in)
• "Timeseries: Timelion chart" (use Timelion icon instead of the clock)
• Sort list of visualizations non-alphabetically to put the most important visualizations first, and to group time-series related items at the bottom.

One thing I don't like right now is that descriptions push visualization types off page. Have we considered getting rid of them? Most people know what an area chart is - we don't need three lines of text explaining it. I personally prefer the simplified view without any descriptions because the chart types don't roll off the screen and Timelion and this new visual builder are a lot more discoverable as a result.

@kimchy @simianhacker @alt74 @cjcenizal @ppisljar @thomasneirynck ^^

[ppisljar]

descriptions: i agree we don't need them .... but not seeing all the charts on one screen .... it really depends more on the number of charts and your screen resolution. What CJ is working on (removing open saved vis from this screen) is gonna solve this problem better. Probably we should put this (removing of description) in the CJs PR and not here.

[ggupta]

In a meeting room with Shay, Chris Cowan, JeffY, Michal, etc and we were going through the features we are highlighting for the Elasticon Keynote. The the naming scheme for some of the chart types in Visualize immediately struck us as a bit odd. 'Timeseries' as a single word is just not a word. It makes it seem like we are coming up with something special, which is not necessary here. Also preceding the two chart types with a colon also seems a little weird (I wonder if there's another way to let people know Timelion is for time series data through help text). A simple proposal is to just use the following for now:

"Time series visual builder"
"Timelion"

[alexfrancoeur]

@simianhacker I think I may have either ran into a bug or am misunderstanding the design. I'm looking at standard stock data from 2009 with other indices on my local kibana instance.

Steps to reproduce:

1. Load up the new vis builder
2. Select Panel Options
3. Change index pattern from * to stocks*. The time field (specific to that index) is trade_date - all is well

5. Change group by from Everything to Terms
6. Terms for all indices appear rather than just those specific to stocks*

However, if I select Aggregate, only the fields specific to stocks* appear. I'd expect the terms to do the same. Let me know if this is a bug or if I'm doing something crazy here.

[simianhacker]

@alexfrancoeur Good catch... fixed

[kimjoar]

Both giving LGTMs and merging while tests are commented out? -1 from me. I'm hoping there is an issue created already to fix this? If we can't get tests to run we have to either fix them or delete them.

[kimjoar]

(Oh, saw the comment in the backport. I still think this should have been fixed first though. Broken windows and all that.)

[simianhacker]

I agree with your sentiment but we (team) wanted to get this commit in this week so it would be available via snapshots for Elastic{ON} and allow more time for people to use it before 5.4 ships. Luckily this is "soft"ware and we can work towards getting Enzyme working again. It broke when the Babel upgrade occurred; had this been merged before that upgrade it would hve been addressed by that team. The issue I have with keeping PR's like this separate for too long is you keep having to chase master as they change things; the PR keeps getting bigger and never seems to finish. With this merged we can now focus on creating smaller more manageable PR's to fix anything that needs addressed between now and code freeze.

[alexfrancoeur]

Update: Issue created for bug tracking #10683
@simianhacker I can't seem to change the date with timelion in both the panel and metric options. They show up as available options but when selected, I get the following error in Kibana. Thoughts?

status: 400, displayName: 'BadRequest', message: '[illegal_argument_exception] Invalid format: "2015-11-25T11:16:25.765Z" is malformed at "T11:16:25.765Z"', path: '/*/_field_stats', query: { level: 'indices', ignore_unavailable: true }, body: { error: { root_cause: [Object], type: 'illegal_argument_exception', reason: 'Invalid format: "2015-11-25T11:16:25.765Z" is malformed at "T11:16:25.765Z"' }, status: 400 }, statusCode: 400, response: '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Invalid format: \\"2015-11-25T11:16:25.765Z\\" is malformed at \\"T11:16:25.765Z\\""}],"type":"illegal_argument_exception","reason":"Invalid format: \\"2015-11-25T11:16:25.765Z\\" is malformed at \\"T11:16:25.765Z\\""},"status":400}', toString: [Function], toJSON: [Function] }

Date format is the following
"dropoff_datetime": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }

There doesn't seem to be an issue in Timelion

[tbragin]

@jimgoodwin Summary & screenshots for 5.4 release blog

Time Series Visual Builder provides a specialized user interface for users working with time series data. It allows users to build up analytics based on time-series in a visual way and uses pipeline aggregations in Elasticsearch to fully distribute all of the computation. While support for some of the pipeline aggregations is available in traditional Visualize charts, with Time Series Visual Builder we wanted to provide a curated experience focused specifically on working with time series data. Making the assumption around time-series simplifies the user interface and saves users many steps when building up these types of charts. Time Series Visual Builder also adds some unique features, such as confidence bands and annotations, specific to the time-series use case. Time Series Visual Builder is marked “experimental” in Kibana 5.4, which means that we may change or modify this feature in the future, and do not guarantee backward compatibility of charts built using this visualization type until it becomes available in GA.

[trevorndodds]

When using derivative with the network counters system.network.in.bytes / system.network.out.bytes when the incrementing counter resets I'm sometimes left with a rather large negative number, this usually occurs when the network is really busy thus increasing the value very quick. How do you deal with these negatives?

[simianhacker]

@trevorndodds Add a "positive only" pipeline agg and set the derivative as the target metric

[trevorndodds]

Great thank you, I had to upgrade.