Geometric interpretation of $\Sigma^{-1}\Phi x$ where $\Phi = [\phi_1,\phi_2,\cdots, \phi_n]$, $\Sigma=\sum_{i=1}^{n} \phi_i\phi_i^T+\lambda I$

Question

Consider $\phi_1, \phi_2, . . ., \phi_n \in \mathbb{R}^d$. Define $\Phi \in \mathbb{R}^{d\times n}$ matrix with $\Phi = [\phi_1, \phi_2, . . ., \phi_n]$ and $\Sigma \in \mathbb{R}^{d\times d}$ by $\Sigma = \sum_{i=1}^{n} \phi_i\phi_i^T + \lambda I$. Let $x \in \mathbb{R}^n$. What is the geometric interpretation of the term $\eta = \Sigma^{-1} \Phi x$, when

1. $\lambda = 0$ and $\phi_1, \cdots, \phi_n$ are linearly independent?
2. $\lambda \neq 0$ and $\phi_1, ..., \phi_n$ are linearly independent?
3. $\lambda \neq 0$ and $\phi_1, ..., \phi_n$ are linearly dependent?

I encountered this $\eta$ while reading a paper in a specific context. There they named the $\eta$ term in a way that suggests some form of projection is involved here. Could someone enlighten me? How to interpret $\eta$ geometrically based on it's construction?

Answer 1

Some potentially useful observations:

• $\sum_{i=1}^{n} \phi_i\phi_i^T = \Phi \Phi^T$
• $\eta$ solves the equation $(\Phi \Phi^T + \lambda I) \eta = \Phi x$
• The $\phi_i$ are linearly independent if and only if $\Phi^T\Phi$ is invertible
• $\Phi\Phi^T$ is invertible if and only if the $\phi_i$ span $\Bbb R^d$. It is only in this case that $\Sigma$ is defined for $\lambda = 0$
• If the $\phi_i$ span $\Bbb R^d$, then there exists a matrix $\Phi$ such that $\Phi \Psi = I_d$. For instance, we can use the Moore-Penrose pseudoinverse $\Psi = \Phi^T(\Phi\Phi^T)^{-1}$. In this case, we can rewrite our equation for $\eta$ as $$ (\Phi \Phi^T + \lambda I) \eta = \Phi x\\ (\Phi \Phi^T + \lambda \Phi\Psi) \eta = \Phi x \\ \Phi(\Phi^T + \lambda \Psi)\eta = \Phi x $$
• When $\lambda = 0$, the equation $\Phi\Phi^T \eta = \Phi x$ describes a least squares solution to $\Phi^T \eta = x$