I've been reading this document. The goal is to find projection of $b$ on the line $L$ which is determined by vector $a$. ($Proj_L(b)$)
In the document, It is mentioned that If $p$ is thought as approximation to $b$, then $e=b-p$ is the error in that approximation (the word approximation is little confusing though, Isn't $e$ the exact value? Although this is not the problem).
Then we know that If $p$ lies on the line through $a$, then $p = xa$ for some $x$. We also know that, $p$ is orthogonal to $e$, therefore their dot product equates to zero:
$a^T(b-xa)=0$
$a^Tb - a^Txa = 0$
$xa^Ta = a^Tb$
$x = \frac{a^Tb}{a^Ta}$
Solving for $p$:
$p = ax = a\frac{a^Tb}{xa^Ta}$
First part was almost completely understandable, but in the second part this projection is written in the terms of projection matrix ("$P: p = Pb$"):
$p = xa = \frac{aa^Ta}{a^Ta}$
where did $b$ go? $x$ and $a$ have changed places, but isn't dot product commutative?
Then, $P$ is solved:
$P = \frac{aa^T}{a^Ta}$
Somehow, $aa^T$ is 3x3 matrix.
How was this concluded? From my knowledge, The general definition of projection matrix is $A (A^{T}A)^{-1} A^T \vec{x}$ (where $A$ is matrix)
Does the definition above has any relations with the projection matrix that was represented in document? If not, how was it derived?
Thank you!
The key point is that from here
$$p = ax = a\frac{a^Tb}{a^Ta}$$
we can write in matrix form
$$p = ax = a\frac{a^Tb}{a^Ta}=\frac{aa^T}{a^Ta}b=Pb$$
From here we can generalize for a projection onto a subspace spanned by multiple vectors $a_i$.
Let consider the matrix $A=[a_1 \, a_2\,...\, a_n]$ and the vector $b$ to project then consider
$$Ax=p$$
the error is $e=b-p=b-Ax$ and it is miminized when $e$ is orthogonal to $Col(A)$ that is
$$A^Te=A^T(b-Ax)=0\implies A^Tb=A^TAx\implies x=(A^TA)^{-1}A^Tb$$
and then
$$p=Ax=A(A^TA)^{-1}A^Tb=Pb\implies P=A(A^TA)^{-1}A^T$$