How to prove that Squared Error Loss is convex

Question

I have the squared loss given by:

$L(z;\textbf{y}) = \frac{1}{2}\left\| z-\textbf{y} \right\|^{2}_{2} = \frac{1}{2}(z-\textbf{y})^{T}(z-\textbf{y})$

where $z=\textbf{W}^{T}\phi(x)+\textbf{b}$.

I need to prove that L is convex with respect to $(\textbf{W},\textbf{b})$. I think it is realted to the Hessian but I can't quite figure out the steps.

Is there any way to prove that does not involve the Hessian?

Answer 1

Fact. Composition of convex function and convex nondecreasing function is convex.

Proof. Let $H$ be a hilbert space. Suppose $g:H \to \mathbb R$ is convex and $f:\mathbb R \to \mathbb R$ is convex and nondecreasing. Set $h := f \circ g$. For any $x,y \in H$ and $t \in [0,1]$, we have $$ \begin{split} h(tx+ (1-t)y) &= f(g(tx+(1-t)y))\\ &\le f(tg(x) + (1-t)g(y))\\ & \le tg(x)+(1-t)g(y)\\ &= th(x) + (1-t)h(x), \end{split} $$ where

• in the second line we have used the fact that $g$ is convex and $f$ is nondecreasing.
• in the third line, we have used the fact that $f$ is convex.

This completes the proof of the above Fact 1.

---

Apply fact one to your problem with $H = \mathbb R^{n \times d} \times \mathbb R^{n}$, $g(W,b) = W^\top \phi(x) + b$ (and affine hence convex function, since hessian vanishes), $f(t) := t^2/2$, a convex nondecreasing function (since 1st and second derivative are nonnegative everywhere).