Convex functions

Course completion
33%
$
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\dom}{dom}
\DeclareMathOperator{\sigm}{sigm}
\DeclareMathOperator{\softmax}{softmax}
\DeclareMathOperator{\sign}{sign}
$

The particular case of convex functions plays an important role w.r.t. to the problem of local minima.

Definition 1.3 (convex function)
Let $f$ be a function defined over some domain $\dom f$. $f$ is said to be convex, iff $$\forall\mathbf{a},\mathbf{b}\in\dom f,\forall k\in[0;1],f(k\mathbf{a}+(1-k)\mathbf{b})\leq kf(\mathbf{a})+(1-k)f(\mathbf{b})$$

This means that any point between $\mathbf{a}$ and $\mathbf{b}$ has an image by $f$ which is below the line segment joining $(\mathbf{a},f(\mathbf{a}))$ and $(\mathbf{b},f(\mathbf{b}))$ as depicted in the figure below.


convex-function
Figure 2: The graph of a convex function $f$. The values of $f$ between two points $a$ and $b$ are always below the chord, i.e. the line segment between $(\mathbf{a},f(\mathbf{a}))$ and $(\mathbf{b},f(\mathbf{b}))$.

Convex functions have the very interesting property that any local minimum is in fact a global minimum, thus simplifying the problem for practitioners.

Next: Continuous differentiable functions