Suppose I have a linear system $y=Ax+\varepsilon$ where $y\in\mathbb R^n$, $x\in\mathbb R^m$ and $A\in\mathbb R^{n\times m}$. I have known that the least squares solution for $x$ is $\hat{x}=A^+y$ where $A^+=(A^HA)^{-1}A^H$ is the pseudoinverse of $R$. However, I have read a book says that the mean square error (MSE) of the solution is (without proof)
$\|\hat{x}-x\|_2^2=\sigma^2\sum_k\frac{1}{\lambda_k}$ where $\lambda_k$ is the $k$-th eigenvalue of the matrix $A^HA$.
Could someone help me how to prove the equation. Thanks a lot!
Assuming that $\varepsilon$ is distributed as $\mathcal{N}(0,\sigma^2 I)$, we have
\begin{align} \text{MSE}=\mathbb{E}\|\hat{x}-x\|_2^2 &=\mathbb{E}\|(A^H A)^{-1}A^H y-x\|_2^2\\ &=\mathbb{E}\|(A^{H}A)^{-1}A^H\varepsilon\|_2^2\\ &=\|(A^H A)^{-1}A^H\|_F^2\sigma^2, \end{align} where the second step uses $y=Ax+\varepsilon$ and the last step can be seen by straightforward calculation. Now, using $\|A\|_F^2=\operatorname{Tr}(A^H A)$, we get \begin{align} \text{MSE}&=\operatorname{Tr}(A^H A)^{-1}\sigma^2\\ &=\sum_i\frac{1}{\lambda_i}\sigma^2 \end{align} where $\lambda_i$ is the $i^\text{th}$ eigenvalue of $A^H A$.