I have heard multiple explanations for what makes something an ORTHOGONAL projection and it's quite confusing.
The first explanation of orthogonal is that if you project b on C(A), the nature of moving the b toward C(A) based on the shortest path is where the orthogonality comes in. But this seems more parallel to me than orthogonal?
The second explanation is that the vector component of b orthogonal to C(A) is the orthogonal part of orthogonal projection. The second explanation makes more sense to me.
But maybe both of those are wrong...
So, a. Anybody have a good explanation for this? b. Is the orthogonal complement equal to the vector of residuals? Is it the same thing?
Let $X$ be a finite-dimensional inner-product space over the real or complex numbers, and let $M$ be a subspace of $X$. Let $x\in X$. The orthogonal projection of $x$ onto $M$ is the unique $m\in M$ such that $(x-m)\perp M$. To see why such an $m$ is unique if it exists, suppose that $m,m'\in M$ satisfy $(x-m)\perp M$ and $(x-m')\perp M$; then $$ ((x-m')-(x-m))\perp M \\ \implies (m-m')\perp M \\ \implies (m-m')\perp (m-m') \\ \implies m-m'=0 \mbox{ or } m=m'. $$ So, the orthogonal projection of $x$ onto $M$ is unique if it exists. To find an orthogonal projection of $x$ onto $M$, choose an orthonormal basis $\{ e_n \}_{n=1}^{N}$ of $M$, and verify that the orthogonal projection of $x$ onto $M$ is $m=\sum_{n=1}^{N}\langle x,e_n\rangle e_n$.