-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide Benchmark.quick_compare to quickly compare methods on an object #134
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not the quick_compare I was hoping for... but not blocking.
lib/benchmark/ips/quick.rb
Outdated
# | ||
# @param warmup How many seconds to warm up the benchmark process | ||
# @param time How many seconds to benchmark each method | ||
def quick_compare(obj, *methods) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please note that in #133 I said nothing about "quickly comparing methods on an object". Please don't make this take an obj
argument that you send on. benchmark-ips is MUCH more than running microbenchmarks on Array to know that reduce
is slower.
Some code needs more complex setup.
Some code is comparing that setup.
Or anything else.
This argument limits things and would require me to make a class with my benchmark methods to get around it. THE POINT of this method is to require less structure to get benchmarks up and running and get an easy comparison of anything.
ary = [Object.new] * 969
ary += [Assertion.new] * 3
ary += [UnexpectedError.new] * 1
ary += [Skip.new] * 27
RESULTS = ary
def report_orig
aggregate = RESULTS.group_by { |r| r.class }
aggregate.default = [] # dumb. group_by should provide this
[aggregate[Assertion].size, aggregate[UnexpectedError].size, aggregate[Skip].size]
end
def report_orig2
aggregate = RESULTS.group_by { |r| r.class }.transform_values(&:size)
aggregate.default = 0
[aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end
def report_tally
aggregate = RESULTS.map { |r| r.class }
aggregate = if RUBY_VERSION > "3.1" then
aggregate.tally(Hash.new 0)
else
aggregate.tally.tap { |r| r.default = 0 }
end
[aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end
def report_tally2
aggregate = RESULTS.map { |r| r.class }
aggregate = aggregate.tally.tap { |r| r.default = 0 }
[aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end
def report_tally3
aggregate = RESULTS.map { |r| r.class }.tally
aggregate.default = 0
[aggregate[Assertion], aggregate[UnexpectedError], aggregate[Skip]]
end
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, you didn't say anything about testing on an object. But without a receiver, who do you want the methods called on? just toplevel methods?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved quick_compare
to Kernel so it's more flexible, more along the lines of your original version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the problem with Benchmark::IPS.quick_compare(self, :report_orig, :report_orig2, ...)
?
Having the object is useful if you have more state you need to pass in.
Then for instance this state could live in an object instance.
Or the methods would be defined as singleton methods of a module, to have the necessary constant scope, etc.
I am against polluting Kernel.
I think a benchmarking harness should not monkey-patch anything.
Also for 10+ years users of this gem managed just fine without this helper method, so maybe we should not add it.
At least I think we should get multiple opinions to not add something which only works for one person.
The existing API is also the same as stdlib Benchmark, it's pretty simple.
I'm back from the dead.
If no one objects I'll pick this up and run w/it |
lib/benchmark/ips/quick.rb
Outdated
# | ||
# @param warmup How many seconds to warm up the benchmark process | ||
# @param time How many seconds to benchmark each method | ||
def quick_compare(*methods) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use regular kwargs here?
def quick_compare(*methods) | |
def quick_compare(*methods, warmup: nil, time: nil) |
lib/benchmark/ips/quick.rb
Outdated
|
||
methods.each do |name| | ||
x.report(name) do |x| | ||
x.times { __send__ name } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about using one of the faster forms, like the explicit while
loop, or string form?
I've changed it to my take.
LMK about objections, otherwise I'll add a test and merge |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love this!
lib/benchmark/ips.rb
Outdated
# def add; 1+1; end | ||
# def sub; 2-1; end | ||
# ips_quick([:add, :sub], warmup: 10) | ||
def ips_quick(methods, receiver = Kernel, **opts) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if having quick in the name would make people think that it runs faster than the normal version
51e724c
to
7024990
Compare
Thanks for everyone's feedback. This should be the final form now, incl. a basic test. If no objections, will merge in a few days. |
module IPS | ||
|
||
# Benchmark-ips Gem version. | ||
VERSION = "2.13.0" | ||
VERSION = "2.13.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just saw the version bump here. Changing to 2.14.
Closes #133
Any feedback welcome, especially from @zenspider