I have the following matrix product:
$\mathbf{B}=\mathbf{A}(\mathbf{I}-\mathbf{P})\mathbf{A}^{\rm T}$,
where $\mathbf{A} \in \mathbb{R}^{n×m}$ is an arbitrary real matrix, $\mathbf{P} \in \mathbb{R}^{m×m}$ is a projection matrix, and $\mathbf{I}$ is the $m×m$ identity matrix. Could you provide a simplified analytical expression for ${\rm det}(\mathbf{B})$ [which is simpler than performing the matrix multiplications in $\mathbf{B}$]?
A projection matrix is always symmetric and therefore diagonalizable.
Therefore $P=S\Lambda S^{-1}$ for some invertible $S$. So, the matrix in question is
$A(SS^{-1}-S\Lambda\,S^{-1})A^{T}=AS(I-\Lambda)S^{-1}A^{T}$. But it is
known that the eigenvalues of $P$ are ones and zeros. We deal with the special case when $A$ is a square matrix. Hence, if there is at least one eigenvalue of $P$ equal to $1$ then
det$AS(I-\Lambda)S^{-1}A^{T}$=$(detA)^{2}det(I-\Lambda)=0$,
so, in general, $det=0$. If all eigenvalues of $P$ are zero's, then
the determinant is equal to $(detA)^{2}$, (since $detA=detA^{T}$).
Now in the general case of a non-square matrix $A$. The result is immediate if all eigenvalues of $P$ are zero and it is $det(AA^{T})$. If all the eigenvalues are $1$ then $det=0$. I am working on the case we have both ones and zeros!
One important special case is when $A$ has full rank. Then $N(A)={0}$ , the nullspace, and all vectors are in $\Re(A^{T}) $, so if there is at least one eigenvalue of $P$ equal to $1$ we get that there is a vector $v\neq 0$ such that $S(I-\Lambda)S^{-1}v=0$ and since $v\in \Re(A^{T})$ there is some non-zero $w$ such that $S(I-\Lambda)S^{-1}A^{T}w=0$ and hence $AS(I-\Lambda)S^{-1}A^{T}w=0$ which implies that the matrix in question has at least one zero eigenvalue and $det=0$.
The result applies to matrices which have rank $r$<$m$. We just write :
$A=[v_{1},v_{2},...v_{r}][I_{r}\,C_{m-r}]L^{T}$ where $L^{T}$ is a permutation matrix and $v_{1},\,v_{2},...,v_{r}$ are the linearly independent columns of $A$. Using the fact that
$\begin{pmatrix} I_{r}\\C_{m-r} \end{pmatrix}$ and $[v_{1}\,v_{2},...v_{r}]$ have full rank=$r$ and L the permutation matrix is invertible and $L^{T}L=I$, we can easily prove that if there is at least one eigenvalue of $P$ equal to $1$ the matrix in question has a nonempty nullspace and therefore has a zero eigenvalue which gives $det=0$.