I'm motivated by a physics research problem to ask whether there's a useful formula for projecting into the null space of the product of two matrices . In particular, if I denote $P_A$ as the projection matrix into the null space of a matrix $A$, I want to find a formula for $P_{AB}$ that might involve calculating simple functions of $B$ like transposes and inverses, but that doesn't involve recalculating a null space.
I came up with two candidates, based on dubious and incorrect arguments:
$C_1 = B^{-1} P_A B$,
$C_2 = B^T P_A (B^T)^{-1}$.
When I multiply these two together, they actually do seem to map the null space to the null space and everything else to zero, but they also mix together the null space vectors in a way I can't unsnarl.
Does anyone know of a way to do this projection? If it helps, the $B$ matrix that I'm interested in is diagonal. I'm particularly interested in something that would be well-behaved in the limit that one of the diagonal entries for $B$ vanishes, making it un-invertible.
Not that it matters for the math, but I'm hoping that if such a formula does exist it will allow us to relate different physically intuitive concepts, and that it might allow us to make efficient calculations on statistical ensembles of systems with the same $A$ but different $B$.
Proof. First of all, it is necessary to state that for any eal $n\times n$-matrix we have \begin{equation}\tag{1} \mathbb R^n = \ker M\,\oplus\operatorname{im}M^*. \end{equation} In other words, $(\ker M)^\perp = \operatorname{im}M^*$. In particular, $I-P_A$ maps onto $(\ker A)^\perp = \operatorname{im}A^*$. The first summand in $R(t)$ is $B^*(I-P_A)$ and thus maps onto $B^*\operatorname{im}A^* = \operatorname{im}(B^*A^*) = \operatorname{im}(AB)^*$. The second summand $tB^{-1}P_A$ maps into $\ker(AB)$ since $AB(tB^{-1}P_A) = tAP_A = 0$.
Assume that $R(t)x = 0$. Then $B^*(I-P_A)x + tB^{-1}P_Ax = 0$. The summands are contained in the mutually orthogonal subspaces $\operatorname{im}(AB)^*$ and $\ker(AB)$, respectively. So, they are orthogonal to each other and must therefore both be zero (see footnote (*) below). That is, $B^*(I-P_A)x = 0$ and $B^{-1}P_Ax = 0$. As $B$ is invertible, we obtain $(I-P_A)x = 0$ and $P_Ax = 0$ and hence $x=0$. Thus, $R(t)$ is indeed invertible.
As $B^*(I-P_A)$ maps to $\operatorname{im}(AB)^* = (\ker(AB))^\perp$, we have $P_{AB}R(t) = tP_{AB}B^{-1}P_A$. But $B^{-1}P_A$ maps into $\ker(AB)$ and thus $P_{AB}R(t) = tB^{-1}P_A$. This gives us $P_{AB} = tB^{-1}P_AR(t)^{-1}$. Since $P_{AB}$ is symmetric, $P_{AB} = P_{AB}^* = (tB^{-1}P_AR(t)^{-1})^* = tR(t)^{-*}P_AB^{-*}$. This proves the proposition.
Maybe you can use the parameter $t$ to control your $B$.
(*) Pythagorean theorem: If $u\perp v$ then $\|u+v\|^2 = \|u\|^2+\|v\|^2$. In particular, if $u+v=0$, then $u=v=0$.