Orthogonal Projections Are Symmetric - Geometric Intuition

Question

Let us denote the projection matrix onto the column space of $A$ by $\pi_A = A(A^T A)^{-1} A^T$. I am looking for geometric intuition as to why it is symmetric. It is very clear to me due to plenty of algebraic reasons (taking transpose, showing $\left \langle \pi_A u,v \right \rangle=\left \langle u,\pi_A v \right \rangle$ and so on...), but I am looking for something of the sort of "proof without words" which could be explained with pictures.

For example, it is clear to me that $\pi_A^2=\pi_A$, since projecting a vector which is already in $\text{col}(A)$ onto $\text{col}(A)$, is itself. Same goes to show $\pi_A A = A$.

I saw many posts addressing this problem, however all the explanations I read resorted to over-killing with calculations.

Answer 1

First let's understand why $\langle x,\pi_Ay\rangle=\langle \pi_A x,\pi_Ay\rangle$. The point is that because $\pi_Ay$ is in the subspace spanned $A$, its inner product with $x$ only "sees" the components of $x$ that lie in that subspace because the components that are orthogonal to it will give an inner product of $0$. More formally, we can write $x= \pi_A x + (x-\pi_A x)$ where the latter term is orthogonal to the subspace (you may check this easily) from which the claim follows by the linearity of the inner product: $$\langle x,\pi_Ay\rangle= \langle \pi_Ax+(x-\pi_Ax),\pi_Ay\rangle=\langle \pi_Ax,\pi_Ay\rangle+\langle x-\pi_Ax,\pi_Ay\rangle=\langle \pi_A x,\pi_Ay\rangle$$

With this in mind, it makes sense that it shouldn't matter if we project $x$ to the span of $A$ first or project $y$ first before taking the inner product, because we are only taking the product "relative to the components in the span of $A$." And this statement about which process we've done first is exactly the statement that $\langle x,\pi_Ay\rangle=\langle \pi_A x,y \rangle$.

Answer 2

Here's an idea you might like. Suppose that $P$ is a projection matrix. That is, $P^2 = P$ but $P$ is not necessarily symmetric. In other words, $P$ is "the projection onto $\operatorname{im}(P)$ along $\ker(P)$". The transpose $P^T$ is another projection; you can verify (by various means) that $P^T$ represents "the projection onto $\ker (P)^\perp$ along $\operatorname{im}(P)^\perp$."

The only time these projections are the same is when $\ker (P) = \operatorname{im}(P)^\perp$. That is, $P$ and $P^T$ can only be the same if $P$ is the orthogonal projection onto $\operatorname{im}(P)$.

---

We can see that $Q = P^T$ must be the projection onto $\ker (P)^\perp$ along $\operatorname{im}(P)^\perp$ as follows. Note that any vector $x$ can be decomposed into $x = x_{im} + x_{\ker}$, where $x_{im} = Px$ and $x_{\ker} = x-Px$.

For any vector $y \in \ker(P)^\perp$ and any $x$, we have $$ \langle y, x\rangle = \langle y, x_{im} + x_{\ker} \rangle = \langle y, x_{im} \rangle = \langle y, Px \rangle = \langle Qy,x \rangle. $$ So, $Qy = y$. Similarly, we can show that $Qy = 0$ for $y \in \operatorname{im}(P)^\perp$.

Answer 3

The orthogonal projection $Px$ of a vector $x$ onto a subspace $M$, is the unique $m\in M$ such that $(x-m)\perp M$. This is equivalent to $$ \langle x-Px,Py\rangle = 0,\;\;\; x,y\in H, \\ \langle x,Py\rangle=\langle Px,Py\rangle $$ Therefore, $\langle x,Py\rangle = \langle Px,Py\rangle$ is symmetric.