Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve benchmarking infra #680

Closed
lorenzwalthert opened this issue Oct 18, 2020 · 1 comment
Closed

Improve benchmarking infra #680

lorenzwalthert opened this issue Oct 18, 2020 · 1 comment

Comments

@lorenzwalthert
Copy link
Collaborator

lorenzwalthert commented Oct 18, 2020

  • The variance in the timing estimates are probably > 1%. A PR and a base reference where no effect on speed is expected (because no such code was touched, Depreciate env var save_after_styling #631), gave a ~2% difference.
  • add time-line plot.
  • artifact should have name of branch when uploaded to later easily compare different artifact.
  • absolute median timing should be posted in PR for easy reference.
  • Need an easy way to show median is x% worse or better than base branch. Currently, we need to estimate from hand and first guess the median. Could be in the plot or posted as comment.
  • Need additional benchmark. Now we have without cache and with cache. The usefulness of with_cache is limited because it is so close to 0 (since it's always full text cached already). We don't have with cache activated and empty cache. This would be helpful for understanding the impact of Speed improvement in caching  #679 for example, or any improvement that relates to the cache. The interpretation of it would include the time required for git reset and can hence not be taken as absolute time to style code, but only in comparison with the base branch (where git reset also occurs).
@lorenzwalthert
Copy link
Collaborator Author

Closed with #698.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant