Using Standard Deviation Aggregation

[angocor]

Hello.

I am trying to create a Datasweet formula that uses a Standard Deviation aggregation as input. However it seems that Datasweet Formula is not capable of using that value.

Is there support for Standard Deviation in Datasweet Formula?.

I am using Kibana and Elasticsearch version 6.2.1 and installed Datasweet Formula datasweet_formula-1.0.1_kibana-6.2.1.zip

[lfroment-datasweet]

Hi @angocor,

indeed, that one creates two columns of data, that is why Formula does not handle it correctly. We need to investigate this.
Regards

[charlotth]

Hi, please try the new release v1.0.2 !

[v01t]

HI @tcharlot-datasweet,

thanks for the update, derivation aggregation works now, same for percentiles, with the following syntax in formula:

• aggX_std_upper
• aggX_std_lower
• aggX_PERCENTILEVALUE

maybe it will be a good improvement (to cover all possible similar cases and functionality) in future to retrieve them in a formula like:
aggX[std_deviation_bounds.upper]
or
aggX[99] (for percentiles)

the reason is that for some of the aggregations elasticsearch return more fields (that currently inaccessible from kibana UI afaik), example with Standard Deviation, response bucket:
... "buckets": [ { "1": { "count": 3, "min": 909, "max": 1670, "avg": 1409.3333333333333, "sum": 4228, "sum_of_squares": 6334382, "variance": 125240.22222222232, "std_deviation": 353.89295305533045, "std_deviation_bounds": { "upper": 2117.119239443994, "lower": 701.5474272226724 } } ...

as you can see bucket beside std_deviation_bounds[] returns also variance, sum_of_squares and others that will be nice to have in the formula, with logic when aggX becoming array accessible from the formula retrieving of such values it is just a matter of proper naming and parentness, like: aggX[sum_of_squares] or aggX[std_deviation_bounds.lower]

I can try to look, maybe I can help with such implementation if you point me on a proper place to do so (to save time).

also, I can create dedicated "feature" issue if required as main subj resolved now.

Regards and thanks a lot for your work.

[charlotth]

Hi @v01t

Actually formula occurs on the array generated by kibana (tabify). This array is based on aggConfig object.
I didnt check but i doubt kibana retrieves the complete values.

For the formula syntax, i agree with you, it's clearer to write aggX[std_deviation]. But, the javascript evaluator doesnt understand this syntax. The following syntax aggX.std_deviation can be possible but in case of percentile values aggX.15will throw an error. That's why i used the following syntax aggX_15.

We will be happy if you want to contribute to this project :)

Regards

[itamarm10]

This plugin is phenomenal! Thank you for your work.
I’m using version 1.1.0 with Kibana 5, and I’m having trouble using the percentile and standard deviation features. When I try to use them on a metric visualization, only “-“ appears.
Perhaps someone can further explain how to use this functionality or upload a screenshot?
Thank you so much!

[lfroment-datasweet]

Hi @itamarm10 ,

because those aggs (percentile and StDev) give you more than one values, you have to write a dedicated Formula input, following this syntax (examples below) :
aggX_std_upper
aggX_std_lower
aggX_PERCENTILEVALUE

[itamarm10]

Thank you very much for your quick response!