Show that when benchmarking properly there's no difference

[domenic]

I was trying to see what micro-optimizations I could make (first few commits). The results were really inconsistent as I reloaded my browser. I added the ratio statistic (fourth commit) to confirm this. It indeed jumped between 0.8x and 100x pretty much at random. So I thought it was time to bring out the big guns and actually use a benchmarking library to get some better statistics.

The fifth commit introduces benchmark.js, and also introduces a check on the .value property (which should avoid JITs optimizing away most of the loop). My results:

Firefox index.html
"sync x 234 ops/sec ±4.25% (11 runs sampled)" test.js:14:2
"promise x 4,217 ops/sec ±19.94% (57 runs sampled)"

Firefox bluebird.html
"sync x 230 ops/sec ±4.00% (11 runs sampled)" test.js:14:2
"promise x 245 ops/sec ±2.73% (79 runs sampled)"

Chrome index.html
sync x 243 ops/sec ±0.76% (8 runs sampled)
test.js:14 promise x 228 ops/sec ±0.70% (33 runs sampled)

Chrome bluebird.html
sync x 242 ops/sec ±0.46% (9 runs sampled)
test.js:14 promise x 243 ops/sec ±0.50% (20 runs sampled)

So if anything promises are faster.

Let me know if you see anything obviously stupid... /cc @petkaantonov

[domenic]

Note that I think the reason benchmark.js is helpful here is because IIUC it runs until the results start settling down. So the fact that we have e.g. 57 runs sampled vs. 11 runs sampled just means that the JIT needed a few more runs to optimize the promise code than it did the sync code.

[petkaantonov]

Why is the sync test async? How much does the result change when you change it to sync?

[domenic]

I made the test async so as to give each of them an equal footing. This is supposed to be a test of how long it takes to do 10 reads in each paradigm, not how many 10-reads you can do in a second. Using different benchmark settings for each result seems like it would give an unfair advantage to the sync one because e.g. if benchmark.js uses setTimeout(,0) in its deferred implementation, that would obviously dominate the results. (Maybe that is what we are seeing here, given that they are so close...)

[domenic]

If we are suspicious of benchmark.js we can try evolving https://github.com/domenic/streams-promise-read/tree/no-benchmarkjs. It currently shows a ~3x slowdown for promises in Chrome, ~18x in Firefox, but I still suspect we haven't properly warmed up the JIT.

[petkaantonov]

Is it possible to benchmark inside a fresh web worker, this would be similar to how node benchmark or bluebird benchmarks do it. Fresh process -> warmup -> do benchmark -> shutdown -> rinse and repeat for next benchmark.

[wanderview]

I haven't looked at the code yet, but this promise result looks completely bogus to me:

Firefox index.html
"sync x 234 ops/sec ±4.25% (11 runs sampled)" test.js:14:2
"promise x 4,217 ops/sec ±19.94% (57 runs sampled)"

[wanderview]

I think the async callback at the end of the sync test is dominating the results with these low numbers of chunks. If you increase the number of chunks to 1000+ you can see the sync case pulling away from promises.

1000 chunks:
"sync x 146 ops/sec ±16.29% (36 runs sampled)" test.js:14:3
"promise x 96.67 ops/sec ±4.96% (68 runs sampled)" test.js:14:3

10000 chunks:
"sync x 40.54 ops/sec ±21.98% (32 runs sampled)" test.js:14:3
"promise x 21.55 ops/sec ±5.12% (53 runs sampled)" test.js:14:3

I guess we can debate the reality of being able to read that many chunks synchronously. (Or wisdom of doing so given frame deadlines, etc.)

I do want to re-run these tests on some mobile devices, though.

[wanderview]

The results in the previous comment were for bluebird. For default Firefox promises... they are much worse:

1000 chunks:
"sync x 119 ops/sec ±27.87% (23 runs sampled)" test.js:14:3
"promise x 26.71 ops/sec ±2.97% (60 runs sampled)" test.js:14:3

10000 chunks:
"sync x 35.56 ops/sec ±20.20% (21 runs sampled)" test.js:14:3
"promise x 2.80 ops/sec ±8.60% (17 runs sampled)" test.js:14:3

[wanderview]

FYI, I opened a bug for firefox's poor performance here: https://bugzilla.mozilla.org/show_bug.cgi?id=1152875

[wanderview]

I haven't looked at the code yet, but this promise result looks completely bogus to me:

Firefox index.html
"sync x 234 ops/sec ±4.25% (11 runs sampled)" test.js:14:2
"promise x 4,217 ops/sec ±19.94% (57 runs sampled)"

I can reproduce this in FF release, but I can't explain the insane number of results we get with the promises here.