Show that that optimal Ridge Regression beta is minimizing

Question

I have the constrained regression (Ridge) of the form $$S(\beta)=(y-X\beta)'(y-X\beta)+\lambda(\beta'\beta)$$ where $\lambda$ is just a scaler. I am able to find an optimizing solution for the beta but I have difficulty proving that the resulting Hessian is positive definite for a minimizing condition to hold. Can you help with that?

Answer 1

The Hessian is $X^TX + \lambda I $ which is trivially positive definite (I'm assuming $\lambda > 0$).

Indeed, for any $u \in \mathbb R^p$, we have $$ u^T(X^TX + \lambda I)u = (Xu)^T(Xu) + \lambda u^Tu \ge \lambda \|u\|_2^2 \ge 0$$ with equality iff $u = 0$.