Let $I_d$ denote the $d \times d$ identity matrix and for $X \in \mathbb{R}^{m\times n}$ and $\lambda>0$ I have to prove the following:
$$(X^TX+\lambda I_n)^{-1}X^T=X^T(XX^T+\lambda I_m)^{-1}$$
I think about using Singular Value Decomposition of $X$, because of the $X^TX$ term, so it holds that:
$$X = U\Sigma V^T$$
But I dont know how to applied correctly.
Thanks in advance.
$\newcommand{\diag}{\operatorname{diag}}$You are on the right track using SVD. Just need to carry some annoying algebra: $$ \begin{align} (X^TX+\lambda I_n)^{-1}X^T&=X^T(XX^T+\lambda I_m)^{-1}\\ \iff (V\Sigma^T U^T U\Sigma V^T + \lambda I_n)^{-1} V\Sigma^T U^T &= V\Sigma^T U^T (U\Sigma V^T V\Sigma^T U^T + \lambda I_m)^{-1}\\ \iff (V(\Sigma^T\Sigma + \lambda I_n)V^T)^{-1} V\Sigma^T U^T &= V\Sigma^T U^T (U(\Sigma\Sigma^T + \lambda I_m) U^T)^{-1}\\ \iff V(\Sigma^T\Sigma + \lambda I_n)^{-1}V^T V\Sigma^T U^T &= V\Sigma^T U^T U(\Sigma\Sigma^T + \lambda I_m)^{-1} U^T\\ \iff V(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T U^T &= V\Sigma^T (\Sigma\Sigma^T + \lambda I_m)^{-1} U^T\\ \end{align} $$ Now all that is left is to show that $(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T=\Sigma^T (\Sigma\Sigma^T + \lambda I_m)^{-1}$. Let us explicitly calculate these matrices:
Without loss of generality assume $m < n$, then $\Sigma^T\Sigma = \diag(\sigma_1^2,\dots,\sigma_m^2, 0,0,\dots)$ and $\Sigma\Sigma^T = \diag(\sigma_1^2,\dots,\sigma_m^2)$. Using the fact that inverse of a diagonal matrix is a diagonal matrix with it's diagonal elements inverted, we have $$(\Sigma^T\Sigma + \lambda I_n)^{-1}= \diag^{-1}(\sigma_1^2+\lambda,\dots,\sigma_m^2+\lambda, \lambda,\lambda,\dots) = \diag((\sigma_1^2+\lambda)^{-1},\dots,(\sigma_m^2+\lambda)^{-1}, \lambda^{-1},\lambda^{-1},\dots).$$ Multiplying on left by the $n\times m$ diagonal matrix $\Sigma^T$, we have $$(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T= \diag_{n\times m}(\sigma_1(\sigma_1^2+\lambda)^{-1},\dots,\sigma_m(\sigma_m^2+\lambda)^{-1}).$$ Note that the extra $\lambda^{-1}$s got killed by the zero rows of $\Sigma^T$.
Similarly for the right hand side: $$(\Sigma\Sigma^T + \lambda I_m)^{-1} = \diag^{-1}(\sigma_1^2+\lambda,\dots,\sigma_m^2+\lambda) = \diag((\sigma_1^2+\lambda)^{-1},\dots,(\sigma_m^2+\lambda)^{-1}).$$ Multiplying on the left by the $n \times m$ diagonal matrix $\Sigma^T$ we have: $$\Sigma^T(\Sigma\Sigma^T + \lambda I_m)^{-1} = \diag_{n\times m}(\sigma_1(\sigma_1^2+\lambda)^{-1},\dots,\sigma_m(\sigma_m^2+\lambda)^{-1}).$$ This concludes the proof.