Create benchmark suite

[MilesCranmer]

This creates a simple benchmark for catching performance regressions on small, tightly controlled problems. To kick things off I added the multivariate first-order optimizers including Adam, AdaMax, BFGS, LBFGS, NGMRES, ConjugateGradient, GradientDescent, and MomentumGradientDescent.

The specific benchmark I added is fitting the Gabor function (2 parameters) to a noisy dataset.

It's compatible with any library supporting BenchmarkTools.jl. You can run the benchmark with:

using Optim
using Pkg
Pkg.activate("benchmark")
include("benchmark/benchmarks.jl")

results = run(SUITE)
@show results

I also add a GitHub action to run AirspeedVelocity.jl on this benchmark for any new PR. It will automatically print out the performance and load time comparison of master in a GitHub comment on the PR.

The current benchmarks across revision history are as follows:

	benchmarks	v1.9.2	v1.8.0	v1.7.6
multivariate/solvers/first_order/AdaMax	0.212 ± 0.00058 ms	0.212 ± 0.00067 ms
multivariate/solvers/first_order/Adam	0.213 ± 0.00058 ms	0.212 ± 0.00054 ms
multivariate/solvers/first_order/BFGS	0.0845 ± 0.00029 ms	0.0843 ± 0.00029 ms	0.0844 ± 0.00025 ms	0.0871 ± 0.00033 ms
multivariate/solvers/first_order/ConjugateGradient	0.273 ± 0.00096 ms	0.273 ± 0.0017 ms	0.274 ± 0.00093 ms	0.291 ± 0.0012 ms
multivariate/solvers/first_order/GradientDescent	0.0842 ± 0.00029 ms	0.0841 ± 0.00029 ms	0.0841 ± 0.00033 ms	0.0868 ± 0.00029 ms
multivariate/solvers/first_order/LBFGS	0.0853 ± 0.00033 ms	0.0851 ± 0.00033 ms	0.085 ± 0.00033 ms	0.088 ± 0.00033 ms
multivariate/solvers/first_order/MomentumGradientDescent	0.0842 ± 0.00033 ms	0.0841 ± 0.00042 ms	0.0842 ± 0.00033 ms	0.0868 ± 0.00029 ms
multivariate/solvers/first_order/NGMRES	0.74 ± 0.0034 ms	0.741 ± 0.0037 ms	0.742 ± 0.0026 ms	0.777 ± 0.0031 ms
time_to_load	0.981 ± 0.009 s	0.99 ± 0.0028 s	0.98 ± 0.02 s	0.364 ± 0.0065 s

Meaning there are no immediate performance regressions. However the load time has clearly increased from v1.7 to v1.8 which #1081 will fix.

---

To create this table above, run the following in bash:

# Install AirspeedVelocity:
julia -e 'using Pkg; pkg"add AirspeedVelocity"; pkg"build AirspeedVelocity"'

# Run benchmark over commit history
benchpkg Optim --rev=benchmarks,v1.9.2,v1.8.0,v1.7.6 -s benchmark/benchmarks.jl --exeflags='--threads=4' --path=.

# Generate the table:
benchpkgtable Optim --rev=benchmarks,v1.9.2,v1.8.0,v1.7.6

We can also get the memory:

	benchmarks	v1.9.2	v1.8.0	v1.7.6
multivariate/solvers/first_order/AdaMax	0.064 k allocs: 2.62 kB	0.064 k allocs: 2.62 kB
multivariate/solvers/first_order/Adam	0.064 k allocs: 2.62 kB	0.064 k allocs: 2.62 kB
multivariate/solvers/first_order/BFGS	0.061 k allocs: 3.44 kB	0.061 k allocs: 3.44 kB	0.061 k allocs: 3.44 kB	0.066 k allocs: 3.83 kB
multivariate/solvers/first_order/ConjugateGradient	0.207 k allocs: 9.09 kB	0.207 k allocs: 9.09 kB	0.207 k allocs: 9.09 kB	0.23 k allocs: 10.9 kB
multivariate/solvers/first_order/GradientDescent	0.054 k allocs: 2.92 kB	0.054 k allocs: 2.92 kB	0.054 k allocs: 2.92 kB	0.059 k allocs: 3.31 kB
multivariate/solvers/first_order/LBFGS	0.083 k allocs: 5.48 kB	0.083 k allocs: 5.48 kB	0.083 k allocs: 5.48 kB	0.088 k allocs: 5.88 kB
multivariate/solvers/first_order/MomentumGradientDescent	0.055 k allocs: 3 kB	0.055 k allocs: 3 kB	0.055 k allocs: 3 kB	0.06 k allocs: 3.39 kB
multivariate/solvers/first_order/NGMRES	0.646 k allocs: 0.0444 MB	0.646 k allocs: 0.0444 MB	0.646 k allocs: 0.0444 MB	0.719 k allocs: 0.05 MB
time_to_load	0.19 k allocs: 17.1 kB	0.19 k allocs: 17.1 kB	0.19 k allocs: 17.1 kB	0.19 k allocs: 17.1 kB

with --mode=memory in the benchpkgtable command. If you want I can put that table in the GitHub action as well?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.79%. Comparing base (78ab1f4) to head (005eb7f).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1084      +/-   ##
==========================================
- Coverage   84.85%   84.79%   -0.06%     
==========================================
  Files          46       46              
  Lines        3480     3480              
==========================================
- Hits         2953     2951       -2     
- Misses        527      529       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pkofod]

Very useful, thanks!