Matrix Inversion distribution

Question

How do you distribute the inversion in $(A^TA+\lambda I)^{-1}A^Ty$ assuming $A$ is a $n \times n$ square invertible matrix, $y$ is a vector with the dimension of $n$, and $\lambda$ is a constant?

Answer 1

Matrix Inversion Lemma:

$$\left(C+DEF \right)^{-1} = C^{-1} - C^{-1}D \left(E^{-1}+FC^{-1}D \right)^{-1} FC^{-1},$$

where $C$, $D$, $E$ and $F$ are matrices of conformable sizes.

---

Applying to your case:

Let $C \equiv \lambda I$, $D \equiv A ^\top$, $E \equiv I$, and , $F \equiv A $, then it follows from the Lemma that

\begin{align} \left(\lambda I+A^\top A \right)^{-1} &= (\lambda I)^{-1} - (\lambda I)^{-1}A ^\top \left(I^{-1}+A(\lambda I)^{-1}A^\top \right)^{-1} A(\lambda I)^{-1} \\[2ex] &=\lambda^{-1} \left(I - A ^\top \left(\lambda I+AA^\top \right)^{-1} A \right) \end{align}

Then, by substituting the result iteratively, it follows that

\begin{align} \left(\lambda I+A^\top A \right)^{-1} A^\top y&=\lambda^{-1} \left(I - A ^\top \left(\lambda I+AA^\top \right)^{-1} A \right)A^\top y\\[2ex] &=\lambda^{-1} \left(I - \lambda^{-1} \left(A ^\top A - \left(A ^\top\right)^2 \left(\lambda I+AA^\top \right)^{-1} \left(A\right)^2\right) \right)A^\top y \\[2ex] &=\lambda^{-1} \left(I - \lambda^{-1} \left(A ^\top A - \lambda^{-1} \left(\left(A ^\top\right)^2\left(A\right)^2 - \left(A ^\top\right)^3 \left(\lambda I+AA^\top \right)^{-1} \left(A\right)^3 \right) \right) \right)A^\top y\\ & \;\;\vdots \\ &= \left(\sum_{j=1}^\infty \lambda^{-j} \left(A^\top \right)^{j-1} \left(A \right)^{j-1} \right)A^\top y + \lim_{j \to \infty} \lambda^{-j} \left(A^\top \right)^{j-1} \left(A \right)^{j-1} A^\top y \end{align}