Randomize layouts #92
Comments
|
The simplest first approach if this seem interesting would be to implement something that randomize the heap allocations. Emery describe a simple approch to that in their paper linked above.
there is also this implementation of it in rust, but it should probably be simpler to do in zig. This could probably be added next to |
|
@FilipAndersson245 thank you for suggesting this. I need to dig a little bit into it. I will have a look 👍 |
|
https://github.com/hendriknielaender/shuffling-allocator.zig Will try to implement this into zbench, if the zig implementation is correct. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
In his talk "Performance Matters", Emery Berger talk about the importance of randomness when doing benchmarks, as a lot of small layout changes may produce large impacts its hard to find when it is an actual improvement. He proposes that if done repeatedly, a complete random layout cannot bias the result. He compares that doing multiple samples over the same layout can be seen as asking 1 person a question 30 times instead of asking 30 people 1 question.
Describe the solution you'd like
Emery Berger brings up these main points of where layout can be randomized to help remove biases.
I do not know how possible all these are in zig, but implementing some of them may help improve benchmark accuracy.
Describe alternatives you've considered
We do not do this and ignore layout optimizations in the code.
Additional context
https://youtu.be/r-TLSBdHe1A?t=950
https://people.cs.umass.edu/%7Eemery/pubs/stabilizer-asplos13.pdf
The text was updated successfully, but these errors were encountered: