orthogonal projection by matrices

Question

I have been trying to understand the theory behind the projection of a vector on a orthonormal bases :-

Suppose {$u_1,u_2,u_3,.......u_n$} be an orthonormal basis for a subspace $W$

Then the $\displaystyle proj_{_W} y=(y.u_1)u_1+(y.u_2)u_2+(y.u_3)u_3........(y.u_n)u_n$

But Here is my question :-If $U=[u_1,u_2.....u_n]$.

Then the $\displaystyle proj_{_W} y=U^TUy$

Since {$u_1,u_2,u_3,.......u_n$} is an orthonormal set, U is an orthogonal matrix

So, $U^TU=I\Rightarrow U^TUy=y$.

Where is the change happening?

Answer 1

Recall that in general the projection matrix is given by

$$P=A(A^TA)^{-1}A^T$$

and for U=A orthonormal we have

$$P=UU^T$$

that is precisely

$$\displaystyle proj_{_W} y=(y.u_1)u_1+(y.u_2)u_2+(y.u_3)u_3........(y.u_n)u_n$$

Answer 2

Your reasoning is mostly correct, but you’ve applied it to the wrong expression. The correct expression for the projection is $UU^Ty$. What you have, $U^TUy$ doesn’t even make sense unless $W$ is the entire parent space: $U^TU$ is an $n\times n$ matrix, but if $W$ is a proper subspace, then $n$ is strictly less than the number of elements in $y$, so doesn’t have the right shape for their product to be defined.

The columns of $U$ are orthonormal, so $U^TU$ is indeed equal to $I_n$, but when $W$ is a proper subspace, $U$ is not square, therefore not orthogonal, so we can’t really say anything about $UU^T$ apart from its size. On the other hand, if $W$ is equal to the entire parent space, then $U$ is orthogonal and $UU^T=I$, but in that case the projection onto $W$ is just the identity map, anyway.

The formula $UU^Ty$ is just the original sum of individual projections onto the basis vectors that you started with, packaged up into matrix form: $$\sum_{i=1}^n (y\cdot u_i)u_i = \sum_{i=1}^n (u_i^Ty)u_i = U \begin{bmatrix}u_1^Ty \\ u_2^Ty \\ \vdots \\ u_n^Ty\end{bmatrix} = UU^Ty.$$