Summary

• An optimization problem consists in finding parameters which minimize or maximize an objective function.
• In the general case, a function can have several local optima or plateaus.
• High dimensional spaces behave in non intuitive ways which can dramatically affect optimization performance.
• Any local optimum of a convex function is in fact a global optimum.
• Local optima can sometimes be found analytically using properties of the gradient and the Hessian.
• Gradient descent is an iterative optimization method which follows the direction of steepest descent at each step and can benefit from high dimensional spaces.
• In black-box optimization the landscape of the objective function cannot be studied analytically and can only be discovered through evaluation.
• Evolutionary algorithms are optimization methods particularly adapted to the black box scenario, which involve moving a population of candidate solutions towards better fitness.
• Evolutionary algorithms allow the practitioner to choose a useful implicit metric with the mutation and cross-over operators.
• EDAs are a mathematically principled form of evolutionary algorithm which move towards better solutions in parameter space by updating a proposal distribution.

Next: From optimization to machine learning