A and B are 2 x 2 matrices with real elements and AB = $A^2B^2 - (AB)^2$ and |B| = 3,
Find the value of |A+2B| - |B+2A|.
Using the given relation, i managed to get |A| = 0 by the following;
AB = $A^2B^2 - (AB)^2$
A = $A^2B - ABA$
A(I - AB - BA) = 0 [ taking determinant on both sides ]
|A||I - AB - BA| = 0
It's here I was stuck on and later on seeing the solution it said this;
" For 2x2 matrices A and B, the following relationship holds true for some scalar x, |A+xB| = |A| + $x^2|B|$ + x[Tr(A)Tr(B)-Tr(AB)] "
Could anyone explain how this relationship was derived or any alternates to solve this problem.
Here is an alternative solution. You have already shown that $A$ is singular. Since $B$ is invertible, from $AB=A^2B^2-(AB)^2$ we obtain $A=A^2B-ABA$. Therefore $A$ has a zero trace. Since $A$ also has a zero determinant, its characteristic polynomial is $x^2$, i.e., it is nilpotent.
Clearly $0\subsetneq\ker(A)\subseteq\ker(BA)$. As the matrices are merely $2\times2$, this means $A$ and $BA$ are simultaneously triangulable. It follows that $|pI+qA+rBA|=|pI+rBA|$ for every real numbers $p,q$ and $r$.
Since $BA$ is singular, one of its eigenvalue is zero and the other is equal to $t=\operatorname{tr}(BA)$. So, we further obtain $$ |pA+qBA+rI|=|qBA+rI|=(qt+r)r=qrt+r^2.\tag{1} $$ Now let $b=\operatorname{tr}(B)$. By Cayley-Hamilton theorem, $B^2-bB+|B|I=0$. Therefore $B^{-1}=\frac{bI-B}{3}$ and $$ \begin{aligned} |xA+yB| &=|B||xB^{-1}A+yI|\\ &=3\left|x\left(\frac{bI-B}{3}\right)A+yI\right|\\ &=3\left|bxA-\frac{x}{3}BA+yI\right|\\ &=3\left(-\frac{xyt}{3}+y^2\right)\quad\text{by }(1)\\ &=3y^2-xyt. \end{aligned} $$ Thus $$ |A+2B|-|2A+B|=(12-2t)-(3-2t)=9. $$