Memory layout differences can render benchmark results meaningless

[kqr]

In the first half of Performance Matters (video) the speaker talks about how big effects memory layout can have on the results of performance benchmarks.

What's worse is that this is a variable that can fluctuate rather wildly for no obvious reason. I don't remember how well the research was made, so that would remain to be vetted.

[hwayne]

We happen to be acquainted, I'll see if he wants to do a writeup

[emeryberger]

I'd say pretty well vetted. Two ASPLOS papers, one by my student @ccurtsinger and myself [ASPLOS'13], reproducing results by from Todd Mytkowicz et al [ASPLOS'09].

https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf
https://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf

[kqr]

Thank you for the references and the great presentation, @emeryberger !

[hwayne]

I'll try to write it up later this month m'self