Low rank SVD, orthogonal projection onto range of A

7.3k Views Asked by At

Say rank(A) = r < n. Then $A = U_{r}S_{r}V_{r}^{T}$. I know the orthogonal projection onto $Ran(A)$ should be $P = U_{r}U_{r}^{T}$ but I'm not sure how to show this. $$ P = A(A^{T}A)^{-1}A^{T} $$

If A had full column rank, then this would be easy since both $V$ and $S$ are invertible. But in this reduced rank case, $S_{r}$ is nonsingular but $V_{r}$ is rectangular and thus, not invertible. So I'm unsure how to simplify the $(A^{T}A)^{-1}$ term...

Also, any hint on how to show that the projection onto $N(A^{T})$ is $P_{N} = \tilde{U}_{r}\tilde{U}_{r}^{T}$? (with $U = [U_{r} \tilde{U}_{r}]$). I thought it would just be $P_{N} = I - U_{r}U_{r}^{T}$

1

There are 1 best solutions below

0
On BEST ANSWER

In this economic version of the SVD of $A\in\mathbb{R}^{m\times n}$ ($m\geq n$), the $U_r\in\mathbb{R}^{m\times r}$ has orthonormal columns, $S_r$ is square diagonal (and nonsingular) and $V_r\in\mathbb{R}^{r\times r}$ is square orthogonal matrix.

It is easy to show, that if $A=WT$, where $W\in\mathbb{R}^{m\times r}$ and $T\in\mathbb{R}^{r\times r}$ is nonsingular, then $\mathrm{Range}(A)=\mathrm{Range}(W)$. To see this, simply consider $x\in\mathrm{Range}(A)$, that is, $x=Ay$ for some $y\in\mathbb{R}^r$. Then $x=Ay=WTy=Wz$ and thus $x\in\mathrm{Range}(W)$. Also, if $x\in\mathrm{Range}(W)$, that is $x=Wz$ for some $z$, then $x=WT(T^{-1}z)=WTx=Ax$, and hence $x\in\mathrm{Range}(A)$. Since $\mathrm{Range}(A)\subset\mathrm{Range}(W)$ and $\mathrm{Range}(W)\subset\mathrm{Range}(A)$, we have $\mathrm{Range}(A)=\mathrm{Range}(W)$.

Now simply set $W=U_r$ and $T=S_rV_r^T$ to get that $\mathrm{Range}(A)=\mathrm{Range}(U)$. Note that $S_r$ is nonsingular and $V_r$ is orthogonal, so indeed, $S_rV_r^T$ is nonsingular as well.

The range of $U_r$ is the range of $A$ and similarly you can show that the range of $P=U_rU_r^T$ is the range of $A$. To see that it is the orthogonal projector, verify that $P^T=P$ and $P^2=P$.

The projector $P$ can be expressed as $P=A(A^TA)^{-1}A^T$ provided that the rank of $A$ is $n$. As above, you can show this by verifying that the range of $P$ defined in this way is the range of $A$ and $P$ is orthogonal projection: $P^T=P$ and $P^2=P$. You can also use your SVD with $r=n$ to see that $$ \begin{split} P&=A(A^TA)^{-1}A^T=U_nS_nV_n^T(V_nS_nU_n^TU_nS_nV_n^T)^{-1}V_nS_nU_n^T =U_nS_nV_n^T(V_nS_n^2V_n^T)^{-1}V_nS_nU_n^T\\ &=U_nS_nV_n^TV_nS_n^{-2}V_n^TV_nS_nU_n^T=U_nS_nS_n^{-2}S_nU_n^T=U_nU_n^T. \end{split} $$

For the rank deficient case ($r<n$), $P$ is given using the Moore-Pseudo pseudo-inverse $A^{\dagger}$ by $P=AA^{\dagger}$. (When $r=n$, $A^{\dagger}=(A^TA)^{-1}A^T$.) There are several (maybe many) ways how to characterize it: it can be, e.g., expressed using the SVD as $A^{\dagger}=V_nS_n^{-1}U_n^T$. Again, you can verify that $P=AA^{\dagger}=U_rS_rV_r^TV_rS_r^{-1}U_r^T=U_rU_r^T$.

For the last question, note that the nullspace of $A^T$ is the orthogonal complement of the range of $A$. If $P$ is the orthogonal projector to the range of $A$, then $Q=I-P$ is the orthogonal projector onto its orthogonal complement, the nullspace of $A^T$. So as you correctly think, $Q=I-U_rU_r^T$. However, the columns of $\tilde{U}_r$ form an orthonormal basis of $\mathrm{Null}(A^T)$. Hence $Q$ can also be expressed as $Q=\tilde{U}_r\tilde{U}_r^T$.

It is quite straightforward to verify that if the columns of the matrix $X\in\mathbb{R}^{m\times r}$ form a basis of some subspace of $\mathbb{R}^m$, the orthogonal projector onto this subspace (that is, $\mathrm{Range}(X)$) is given by $X(X^TX)^{-1}X^T$. If $X$ has orthonormal columns, (that is, $X^TX=I$), the projector is simply $XX^T$. Since $\tilde{U}_r$ is the orthonormal basis of the orthogonal complement of the range of $A$, $Q=\tilde{U}_r\tilde{U}_r^T$.