Benchmarking scripts for paper and continuous benchmarking

[Markus-Goetz]

Implemented and unified benchmarks for the paper. First step towards continuous benchmarking by providing benchmark configurations, jobscript generation and optional batch submission. Actual implementation in line with PRs/releases still missing.

Description

Issue partially resolved: #466

Changes proposed:

• Added statistical moments benchmark
• Added distance matrix benchmark
• Added kmeans benchmark
• Added lasso benchmark
• Added configuration files for each benchmark
• Added jobscript generator with optional submission to batch system

Type of change

• New feature (non-breaking change which adds functionality)

Due Diligence

• All split configurations tested
• Multiple dtypes tested in relevant functions
• Documentation updated (if needed)
• Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

[mtar]

Can one of the admins verify this patch?

[coquelin77]

need to either have everything as floats or as doubles in np and dask, as well as in heat

need 2 processes for CPUs

perf counters not started everywhere. are we using these? it doesnt work for dask anyway

[codecov]

Codecov Report

Merging #652 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #652   +/-   ##
=======================================
  Coverage   97.37%   97.37%           
=======================================
  Files          85       85           
  Lines       16646    16646           
=======================================
  Hits        16209    16209           
  Misses        437      437           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82c0df5...76342e8. Read the comment docs.

[Cdebus]

There are some things still not entirely correct