Projection on to a Subspace and orthogonal basis confusion

Question

https://www.cliffsnotes.com/study-guides/algebra/linear-algebra/real-euclidean-vector-spaces/projection-onto-a-subspace

I am following this example here. It is written in a way that clarifies some things I didn't quite grasp before, however, this part, I don't quite understand

"The vector $v_{\parallel S}$, which actually lies in $S$, is called the projection of $v$ onto $S$, also denoted $\text{proj}_S v$. If $v_1, v_2, \dots, v_r$ form an orthogonal basis for S, then the projection of $v$ onto $S$ is the sum of the projections of $v$ onto the individual basis vectors, a fact that depends critically on the basis vectors being orthogonal:"

a. That bolded part is especially unclear. First, can you project on to subspaces, like S, that DO NOT have an orthogonal basis? b. not understanding what they mean by the projection of v onto S is the SUM of the projections of v on to the INDIVIDUAL basis vectors

Regards

Answer 1

When it says, "projections ... onto the individual basis vectors," that's a little bit sloppy; what it actually means is, projections onto the subspaces generated by the individual basis vectors. One projects onto a subspace, not onto a vector.

Now, you ask about subspaces that don't have an orthogonal basis. There is no such thing. All this is taking place in some (bigger) vector space $V$, and $V$ has to be an inner product space, else there isn't any such concept as projection; and since $V$ is an inner product space, so are each of its subspaces, and each subspace has an orthogonal basis, as shown by the Gram-Schmidt construction (well, at any rate if we're talking about finite-dimensional spaces, that works).

But maybe you meant subspaces for which the only basis you happen to know is not an orthogonal basis. The concept of a projection onto such a subspace is still valid since, as noted already, you can transform your non-orthogonal basis into an orthogonal basis. Even if you don't know an orthogonal basis, and even if you don't go get one, you can still compute the projection of any given vector onto the subspace; the formulas are a bit more complicated than they are in the orthogonal basis case, and are usually not presented in Linear Algebra classes.

Finally, if you do have an orthogonal basis, and if for each member of that orthogonal basis you compute the projection of $v$ onto the subspace spanned by that member, and then you add up all those projections, you get the projection of $v$ onto the subspace. That should answer your (b) question.

Answer 2

Take for example, the set $S$ to the plane spanned by $v_1 =(1, 2, 1) $ and $v_2 = (-1, 0, 1)$ which are two orthogonal vectors. And let the vector to projected onto $S$ be $ v = (2, 3, 4) $. Let $w$ be the projection of $v$ onto $S$, then in matrix-vector form, $ w = A x = v + p $

where $p$ is orthogonal to both $v_1 $ and $v_2$

where $A = [v_1, v_2]$

Premultiplying by $A^T $ (the transpose of matrix $A$ ) we obtain

$A^T A x = A^T v + A^T p = A^T v $

Since the two columns of $A$ are linearly independent, $A$ has full rank and thus

it is invertible, and we have,

$x = (A^T A)^{-1} A^T v $

And finally the vector $w = A x = A (A^T A)^{-1} A^T v$

If the two columns of $A$ are orthogonal, then $A^T A$ is diagonal, and we have

$x_1 = \dfrac{v_1^T v}{v_1^T v_1} $ and $ x_2 = \dfrac{v_2^T v }{v_2^T v_2} $

from which,

$w = A x = x_1 v_1 + x_2 v_2 $

Note that $x_1 v_1 $ and $x_2 v_2 $ are just the projections of $v$ onto the subspaces: Span{$v_1$} and Span{$v_2$}.

The above verifies the statement made in the quoted paragraph.

So for our $v_1 $, $v_2$, and $v$, the projection is simply

$w = \dfrac{12}{6} (1, 2, 1) + \dfrac{2}{2} (-1, 0, 1) = (1, 4, 3) $

If $v_2 $ is replaced by another basis vector like $v_3 = (0, 2, 2)$ then

$v_1$ and $v_3$ are no longer orthogonal, thus we cannot say that

$ w = \dfrac{12}{6} (1, 2, 1) + \dfrac{14}{8} (0, 2, 2) =$ (wrong answer)

However, what we can do is follow the above procedure, by defining

$A = \begin{bmatrix} 1 && 0 \\ 2 && 2 \\ 1 && 2 \end{bmatrix}$

Then it would follow that $A^T A = \begin{bmatrix} 6 && 6 \\ 6 && 8 \end{bmatrix} $

and $(A^T A)^{-1} = \dfrac{1}{12} \begin{bmatrix} 8 && -6 \\ -6 && 6 \end{bmatrix}$

and the projection is given by

$w = A (A^T A)^{-1} A^T v $

we have $A^T v = \begin{bmatrix} 1 && 2 && 1 \\ 0 && 2 && 2 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 12 \\ 14 \end{bmatrix}$

And therefore,

$ (A^T A)^{-1} A^T v = \dfrac{1}{12} \begin{bmatrix} 8 && -6 \\ -6 && 6 \end{bmatrix} \begin{bmatrix} 12 \\ 14 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} $

Hence, finally,

$ w = A x = v_1 + v_3 = (1, 4, 3) $ (correct answer)