Transpose of a projection matrix

Question

The projection matrix for some matrix $A$, which has $m>n$ is:

$$P = A(A^TA)^{-1}A^T$$

A property of the projection matrix is that: $$P^T=P$$

If I call $(A^TA)^{-1}=B$, then:

$$P=(AB)A^T$$

Now I take transpose: $$P^T = ((AB)A^T)^T=A(AB)^T=AB^TA^T$$ or $$P^T=A((A^TA)^{-1})^TA^T$$

And is this equal to $P$? That middle term is transposed...

Answer 1

Remember that transposition and inversion commute, i.e. the transpose of the inverse is equal to the inverse of the transpose: $$\left(B^\mathrm{T}\right)^{-1} = \left(B^{-1}\right)^\mathrm{T}.$$ Using this fact, we have $$\left(\left(A^\mathrm{T}A\right)^{-1}\right)^\mathrm{T} = \left(\left(A^\mathrm{T}A\right)^{\mathrm{T}}\right)^{-1} = \left(A^\mathrm{T}A\right)^{-1},$$ where the last equality follows since $A^\mathrm{T}A$ is symmetric.

Answer 2

The inverse and the transpose operations commute: assuming $A$ is square, $(A^T)^{-1}=(A^{-1})^T$, because

$$A^T((A^{-1})^T)=(A^{-1}A)^T=I^T=I$$

Answer 3

If $A$ has full column rank, then its SVD is of the form

$$A = U \Sigma V^T = \begin{bmatrix} U_1 & U_2\end{bmatrix} \begin{bmatrix} \hat{\Sigma}\\ O\end{bmatrix} V^T$$

Thus,

$$A^T A = V \Sigma^T U^T U \Sigma V^T = V \Sigma^T \Sigma V^T = V \hat{\Sigma}^2 V^T$$

and

$$(A^T A)^{-1} = V \hat{\Sigma}^{-2} V^T$$

Hence,

$$A (A^T A)^{-1} A^T = U \Sigma V^T V \hat{\Sigma}^{-2} V^T V \Sigma^T U^T = U \Sigma \hat{\Sigma}^{-2} \Sigma^T U^T = U \begin{bmatrix} I & O\\ O & O\end{bmatrix} U^T = U_1 U_1^T$$

which is symmetric. The columns of $U_1$ form an orthonormal basis for the column space of $A$.

Answer 4

Yes, $P^T$ is equal to $P$ because $(A^T A)^{-1}$ is equal to its transpose. This is because for any invertible matrix $X$, $(X^T)^{-1} = (X^{-1})^T$, ie the operations of taking inverses and transposes commute. So we have that the transpose $((A^TA)^{-1})^T = ((A^TA)^T)^{-1}$, which equals $ (A^T A)^{-1}$ because $(A^TA)^T = A^TA$.

To prove that taking inverses and transposes can be done in either order, suppose $X$ is a matrix with inverse $X^{-1}$, so that $X X^{-1}=I$. Taking the transpose of both sides, we get $(X^{-1})^TX^T=I$. In other words, $(X^{-1})^T$ is the inverse of $X^T$, i.e. $(X^{-1})^T = (X^T)^{-1}$.