How to factor a matrix in a multiplication of 4 matrix?

Question

Quick question but nonetheless very needed, let's say we have the expression $AB^T AB^T+AB^T$ and we want to factor out the $AB^T$, being $A,B$ matrices. Would it be $AB^T(AB^T+\mathbb{I})$ or would it be $AB^T(B^T A+\mathbb{I})$?

Answer 1

Hints: matrix multiplication is not commutative except in the very special case of $1 \times 1$-matrices. Useful matrices to think about are the matrices that with exactly one non-zero entry: a standard notation for these is to write $\Delta_{ij}$ for the matrix with all entries $0$ except for a $1$ in row $i$ and column $j$. Looking at 2-dimensional examples is often a good place to start as it makes the calculations easy. If you take $A = B = \Delta_{12} = \Delta_{21}^T = \pmatrix{0 & 1 \\ 0 & 0}$ in your example, I think you will find that:

$$AB^T(AB^T + \Bbb{I}) = AB^TAB^T + AB^T =\pmatrix{2 & 0 \\ 0 & 0}$$

while $$AB^T(B^TA + \Bbb{I}) = \pmatrix{1 & 0 \\ 0 & 0}$$

It is also true that $AB^TAB^T + AB^T = A(B^TA + \Bbb{I})B^T$ as you conjectured in your comment. The associativity, distributivity and unit laws can be used to prove the true assertions in the general case.