Statistically compare benchmark results #21
Labels
discussion
Talking about something in order to reach a decision or to exchange ideas
enhancement
New feature or request
Comparing min/avg etc. values is nice but can be misleading.
I propose to implement a separate script for comparing k6 summaries (extend them if needed), similar to benchstat .
Basically, it should be obvious for a performance engineer what improvement the change in code produces.
As an example, here is benchstat output:
We can see a deviation from the mean as well the change in the second benchmark being statistically insignificant.
The only difficulty I see is that we might need to store results for all operations in the benchmark. Still possible.
I believe automated regression tests could also use such feature.
cc @anikeev-yadro @jingerbread
The text was updated successfully, but these errors were encountered: