No standard way of testing performance

[fweik]

We need a way to track performance changes by pull requests.

[jngrad]

2018 update: introduction of a benchmarking feature (#2381), needs to be run manually
2019 update: the feature is now more automatized (#2998) and re-uses some logic from the CI script

There are still stability issues (#2924). The Python scripts in the benchmarks are tested (887597b) but not the CMake and bash logic. The benchmarks are documented here. Tracking performance by PR will require a dedicated machine with no other concurrent job to limit interference.