Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get and report return value of benchmark functions #9

Open
eriknw opened this issue Jun 21, 2014 · 0 comments
Open

Get and report return value of benchmark functions #9

eriknw opened this issue Jun 21, 2014 · 0 comments

Comments

@eriknw
Copy link
Member

eriknw commented Jun 21, 2014

I would love for benchmark functions to be able to return a value. This can be used for several things:

  1. User can test benchmark functions
  2. Verify that the returned value is consistent for each implementation variant
  3. If specified, perhaps the output of the benchmark functions can be given

It is good practice for the user to provide a reference implementation of the function in the benchmark file. For example, if toolz.groupby is being benchmarked, this is as simple as doing from toolz import groupby in the benchmark file. Another option is to define it in the benchmark file, such as def groupby(key, seq). The reference implementation isn't benchmarked, but it does allow benchmark functions to be tested.

If no reference implementation is given, the output of each variant can still be compared to make sure they are all equivalent. If there is a reference implementation, then it is possible to identify which variants are incorrect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant