Proof that loss function for linear regression is an ellipsoid

Question

I cannot seem to find a proof that $f(\mathbf{\beta}) = \left\lVert \mathbf{y}-\mathbf{X} \mathbf{\beta} \right\rVert^2$ is an ellipsoid, centered at the OLS solution $\hat{\beta}$. Can anyone show how to convert it to the quadratic form of a general ellipsoid, i.e. $(\beta - \hat{\beta})^T \mathbf{A} (\beta - \hat{\beta})$, where $\mathbf{A}$ is positive definite?

Answer 1

The function $(\beta - \hat{\beta})^\top A (\beta - \hat{\beta})$ is zero when $\beta = \hat{\beta}$, yet $f(\hat{\beta}) = \|y - X \hat{\beta}\|^2$ might not be zero.

The form you are looking for is $(\beta - \hat{\beta})^\top A (\beta - \hat{\beta}) + c$ for some vector $c$ (which will turn out to be $\|y - X \hat{\beta}\|^2$). This shows the level sets of $f$ are ellipsoids centered at $\hat{\beta}$.

---

$\hat{\beta}$ is characterized by $$X^\top y = X^\top X \hat{\beta}.\tag{$*$}$$

Using this we have $$ \begin{align} (\beta - \hat{\beta})^\top X^\top X (\beta - \hat{\beta}) &= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top X \hat{\beta} + \hat{\beta}^\top X^\top X \hat{\beta} \\ &= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top y + \hat{\beta}^\top X^\top y \\ &= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top y+ y^\top y - y^\top(y - X\hat{\beta}) \\ &= \|y - X \beta\|^2 - y^\top (y - X\hat{\beta}). \end{align}$$

So, $f(\beta) = (\beta - \hat{\beta})X^\top X (\beta - \hat{\beta}) + y^\top (y - X\hat{\beta})$. Note that the additive term $y^\top (y - X \hat{\beta})$ is constant (since it does not depend on $\beta$), and can also be written as $\|y - X \hat{\beta}\|^2$ using the normal equation ($*$).

$$f(\beta) = (\beta - \hat{\beta})X^\top X (\beta - \hat{\beta}) + \|y - X \hat{\beta}\|^2.$$