Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regularize mandelperf timings to include sums & assertions or not #10

Open
johnfgibson opened this issue May 21, 2018 · 2 comments
Open

Comments

@johnfgibson
Copy link
Contributor

The timings of the mandelbrot calculations are not consistent between all languages. In some (C, Rust), the timings include summing the total number of iterations over the specified grid and asserting that number is 14791. In others (Python, Julia), the timings only include iterating over the grid, not the sum or assertion. Some languages repeat mandelperf 100 times and return total time / 100, others return the time for a single run.

@StefanKarpinski
Copy link
Member

StefanKarpinski commented May 21, 2018

IIRC, the languages where we only time a single run are the ones where that run takes as long or longer than 100 runs of faster languages. Feel free to make it more consistent but I’d really rather not have to wait for several hours while Octave runs the benchmarks 100 times.

@Enet4
Copy link
Contributor

Enet4 commented May 23, 2018

I noticed that parse_integers might also have similar inconsistencies: The Julia version checks the outcome of parse only at the end of t iterations (line 26, and Lua too), but the C version... presumably it would check every single parsed number (line 263), but asserts made with assert can be cleared out in compile time if the NDEBUG flag is set. I don't think this is the case, but I can double-check that. Just checked, the asserts are there regardless of the optimization level.

As the Rust version was influenced by C's and Julia's, it checks all parsed numbers unconditionally (line 278).


Update: this has been addressed in #18. The check in parse_int should indeed happen inside the loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants