****optimize nelder_mead

Description

The nelder-mead or simplex method ([U13]) is an order 0 local optimizer. The method is numerically simple and robust, but slow for large scale problems. A flow chart of the method is given in the figure hereafter. The simplex method starts with a regular geometric figure called the simplex consisting of \(n+1\) vertices where \(n\) is the dimension of the design space. Then, vertices are moved according to their respective values \(f\). There are 3 main steps during a sequential simplex search: the reflection (from the worst point to the best), the expansion (continuation of the displacement towards the best vertex, when reflection improved \(f\)), and the contraction of all the vertices towards the simplex gravity center when reflection has failed. In the figure, xh, xl, xs and xm are the highest, lowest, second lowest point, and the gravity center, respectively. fh, fl, and fs are associated cost functions. Typically, \(a~\approx~ 1\), \(b~\approx~2\), et \(0~<~c~<~1\).

This implementation integrates an optional automatic restarting procedure in an attempt to render the algorithm less sensitive to local minima. In this case, the new starting point is automatically evaluated based upon previous results. Optimization constraints are also supported, by means of a basic penalty technique.

Syntax

The nelder-mead optimizer has a number of adjustable parameters which can be set in the ***convergence section. These are summarized below.

tolerance

Minimal relative difference between the highest and lowest vertex for convergence. This is exclusive with min. The algorithm stops when the last change in objective function is fractionally smaller than tolerance. It is the default stopping criterion with a value of 1.e-9.

size_optimum

Minimal value of the objective function below which the search stops. This is exclusive with tolerance. It is not the default stopping criterion. Default value is 1.e-5.

size_degeneration

Minimal relative value below which, simplex is considered as degenerate, ie. vertices are no more linearly independent. Default value is 1.e-6.

length

Initial perturbation magnitude as a factor times the initial value. It is the characteristic length scale. It is used to construct the initial simplex from one starting point. Default value is 1. It is a rather sensitive parameter, and one should play with it if the simplex does not start the search properly.

iter

Maximum function evaluations before terminating. Default value is 100.

max_restart

Maximum number of restarts. Default value is 0 (ie. no restart).

reflection

Value of the reflection parameter a (default value is 1.0)

expansion

Value of the expansion parameter b (default value is 2.0)

contraction

Value of the contraction parameter c (default value is 0.5).

lag_mult_step

Step size used to update the penalty parameters associated to the optimization constraint: \(if ~g_i>0. ~~then~~ \lambda_i += step~.~g_i\). Default value is 0.01.

constraint_tol

Constraint violation tolerance value, ie. constraints are verified when \(g_i(x)<constraint\_tol\), and the corresponding design point \(x\) retained as feasible. Default value is 0.001.