Suggestions for optimization algorithms to test

[sgbaird]

design of experiments (DOE)

(suggestion by @kjelljorner, see twitter post)

• Latin Hypercube is a common one (implemented in Olympus, see below). See also DOEPY, EDIT: also suggested by Tonio Buonassisi
• MINITAB summary of a few different DOE methods
• DESICE and JMP

L-BGFS

(suggestion by @CompRhys, see twitter post)

• can use Olympus (see below), see also scipy's L-BGFS-B

derivative-free method, e.g. simplex

(suggestion by @WardLT, see twitter post)

• can use Olympus (see below), see also scipy's linprog

Olympus

Supports various optimization algorithms in addition to benchmarks (suggestion by Alan Aspuru-Guzik, see twitter post, cc @rileyhickman). Includes many options from above.

Supported algorithms in Olympus

From Colab notebook

['BasinHopping',
 'Cma',
 'ConjugateGradient',
 'DifferentialEvolution',
 'Genetic',
 'Gpyopt',
 'Grid',
 'Hyperopt',
 'LatinHypercube',
 'Lbfgs',
 'ParticleSwarms',
 'Phoenics',
 'RandomSearch',
 'Simplex',
 'Slsqp',
 'Snobfit',
 'Sobol',
 'SteepestDescent']

Olympus also has a nice page categorizing the algorithms into "Bayesian", "Evolutionary", "Gradient", "Grid-like", and "Other". This is right in line with a results figure I'm thinking about that using plotly legend groups (see group click toggle behavior).

[WardLT]

A quick note: The "simplex" I was referring to is this one - https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html#optimize-minimize-neldermead. It could be the same as the "Simplex" mentioned in Olympus.

This simplex is a local optimizer, but you can make it "global" by just running it with different starting guesses (i.e., 'multi-start' optimization).

[sgbaird]

@WardLT thanks for clarifying! Looks like Olympus also uses Nelder Mead via scipy.

[kjelljorner]

@sgbaird Not really an expert on DoE myself. People seemed to use proprietary software like MODDE and also in an iterative manner depending on how many experiments could be run in a batch etc. I'm not sure if it is possible to reproduce that workflow with open-source tools and how easy it would be to automate.

[sgbaird]

I think Latin Hypercube Sampling might be a good option for DOE. Might try to use one of the proprietary softwares as well.

A visualization that I'm interested in is grouping the algorithms into categories (as mentioned above) and making it an interactive Plotly figure with legend groups and the ability to toggle individual traces.