"Perturbed projection" matrix equality

Question

I recently came across a throwaway comment in a paper noting the following equality, for an $n\times p$ matrix, $X$, and $\lambda > 0$:

$$I_n - X(X'X + \lambda I_p)^{-1}X' = \lambda(X X' + \lambda I_n)^{-1}.$$

Computing this (symbolically) for small $n, p$ and testing it numerically for various randomly drawn matrices $X$ has mostly convinced me that this does indeed hold in general, however I have thus far been unable to prove this.

A proof, or - ideally - any hint as to how to show this would be greatly appreciated. (I have a feeling this is going to be one of those where I end up - metaphorically - kicking myself when I see the solution...).

Answer 1

Here's a suggestion (not sure if it will work): For fixed $X$, consider the function $$f(\lambda) = I_{n} - X(X'X + \lambda I_{p})^{-1}X - \lambda (XX' + \lambda I_{n})^{-1}$$ from $\mathbb{R}$ to $\mathbb{R}^{n^{2}}$. Presuming it is smooth (or even maybe analytic), taking derivatives at a point may allow you to claim that it is constant. Evaluating at a point, $\lambda = 0$ would then show that it is zero everywhere, thus implying the equality for all $\lambda$.

Edit: on second thought, $f$ is not necessarily defined for $\lambda = 0$, but rather only for $\lambda$ sufficiently far from zero (if $X'X$ is not invertible).

Answer 2

Let $UDV'$ be the singular value decomposition of $X$. In particular, if $q = \min(n, p)$ we have $X = UDV'$ where $U$ is a $n\times q$ matrix, $D$ a $q\times q$ diagonal matrix and $V'$ a $q\times p$ matrix, where both $U$ and $V$ are orthogonal. After some algebra, and noting that $V^{-1} = V'$, it follows that $X(X'X + \lambda I_p)^{-1}X' = UD(D^2+\lambda I_q)^{-1}DU'$. Similarly, noting that $U^{-1} = U'$, we have $\lambda(XX' + \lambda I_n)^{-1} = \lambda U (D^2 + \lambda I_q)^{-1}U'$. Combining these, we have

\begin{align} I_n - \left[X(X'X + \lambda I_p)^{-1}X' + \lambda(XX' + \lambda I_n)^{-1}\right] &= I_n - \left[UD(D^2+\lambda I_q)^{-1}DU' + \lambda U (D^2 + \lambda I_q)^{-1}U'\right]\\ &= I_n - U\left[D(D^2+\lambda I_q)^{-1}D + \lambda(D^2 + \lambda I_q)^{-1}\right]U'\\ &= 0. \end{align}