I am learning multivariable calculus, and I want to make sure my way of solving is correct.
I want to prove if this equation is a convex by checking the hessian (second derivative). Can someone help correct me if I am wrong?
$$\phi : \theta \mapsto \vert\vert X\theta - Y\vert\vert^2 = \vert\vert X\theta\vert\vert^2 - 2X\theta Y + \vert\vert Y\vert\vert^2$$ $$\frac{\partial\phi}{\partial\theta}= 2(X\theta)'X - 2XY$$ $$\frac{\partial ^2\phi}{\partial \theta^2} = 2\vert\vert X\vert\vert^2$$
which is a positive semi-definite matrix, since $2\vert\vert X\vert\vert^2 \geqslant0$. Thus, $\phi$ is convex.
You're almost there, except for an error in what you wrote. If $X$ is an $n$-by-$p$ matrix with $p \ne 1$, then $X'X$ is a $p$-by-$p$ matrix, not the scalar $\|X\|^2$ (norm of $X$ squared) ! The right expression for the Hessian is $\partial^2 \phi = 2X'X$ which is psd since $u'X'Xu = ||Xu||_2^2 \ge 0$ for all $u$.