Minimal viable benchmarks

[sheplu]

Following the discussion during the first meeting of the team a few topics were raised regarding benchmarks

• how we want to benchmark the runtime
• what should we benchmark

Some issues were also discussed about benchmarks

• time to run all benchmark
• in which architecture
• how frequently

I will grab and put the discussion we had yesterday (at least the key points made) so we can have a constructive talk around it

[anonrig]

We should keep in mind that certain benchmarks take a lot of time to execute right now. For example, running all URL benchmarks take 4 hours. (Referencing https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1222/)

[BethGriggs]

nodejs/Release#479 is old but relevant. If we had a small set of benchmarks that provided a reasonable amount of coverage/value, then we could consider running them as part of the release process (for specific, or even all releases).

The caveat is that they would need to take a similar amount of time as our builds and be reasonably easy to interpret to avoid adding too much time/effort to the already laborious release process. Maybe that could initially be achieved by setting a generous max regression % that if we go below, then we should investigate before shipping the release.

[sxa]

@sheplu I didn't manage to get on the call yesterday but it sounded from the recording like you had information from quite a few sources - are there more reference links that you can add into this issue so we can use this as an initial "hub" for collating information about what we've currently got?

[sheplu]

I was talking a bit linked to this issue #13

Here is the interesting links

• https://github.com/nodejs/benchmarking/tree/master/benchmarks
• https://github.com/mscdex/nodebench.
• https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1222/

Now that I list them here I was imagining more resources :D

Also a good talk to have is even if we don't benchmark everything every times we should have something to test some of the core feature on some "basic" use case