Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report standard deviation in benchmarks #7927

Open
2 tasks
Akirathan opened this issue Sep 29, 2023 · 3 comments
Open
2 tasks

Report standard deviation in benchmarks #7927

Akirathan opened this issue Sep 29, 2023 · 3 comments
Assignees
Labels
-compiler -tooling Category: tooling x-new-feature Type: new feature request

Comments

@Akirathan
Copy link
Member

Our benchmark charts currently displays only score which is an average of one benchmark iteration in milliseconds.

Since JMH can, and does, output stddev, it would be nice to:

Tasks

Preview Give feedback

stddev is a very important metric as it tells us basically how much we can trust the results for a particular benchmark.

@Akirathan Akirathan added -tooling Category: tooling x-new-feature Type: new feature request labels Sep 29, 2023
@Akirathan Akirathan self-assigned this Sep 29, 2023
@radeusgd
Copy link
Member

image

As noted, the std-dev allows us to easier differentiate benchmarks like the navy vs orange ones from this analysis (one has very high variance and the other is very 'stable').

@radeusgd
Copy link
Member

Just a note that in #8091 I added displaying stdev within the Enso runner for the benchmarks. This is a bit perpendicular, because this is not the runner used on CI so the ticket remains open - but it is slightly related so I thought it may be worth noting.

@radeusgd
Copy link
Member

image
We can see how the stdev allows us to better judge the benchmarks - for example the high stdev in the Enso variants may show that the warmup time is insufficient and should be made larger if we want to see the peak performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-compiler -tooling Category: tooling x-new-feature Type: new feature request
Projects
Status: New
Development

No branches or pull requests

3 participants