Let $A$ and $B$ be matrices of order $3\times 2$ and $2\times 3$ respectively. Suppose that their product $AB=\begin{bmatrix} 8 & 2 &-2\\ 2 & 5 &4\\ -2 & 4 &5\\ \end{bmatrix}$ and $BA$ is non-singular, then find $det(BA)$
My Attempt
Considering the order of the given matrices according to established theory $det(AB)=0$.
But now how to find $det(BA)$
the most straightforward answer would be to use Cauchy-Binet or Newton's Identities.
if you don't know Newton's Identities, you can get to the same result here via Cayley Hamilton
$C:=BA$
per Cayley Hamilton
$C^2 - \text{trace}\big(C\big)C + \det\big(C\big)I_2 = \mathbf 0$
re-arranging terms
$ \det\big(C\big)I_2 = -C^2 + \text{trace}\big(C\big)C $
taking the trace and dividing by 2
$ \det\big(BA\big) = \det\big(C\big) = -\frac{1}{2}\text{trace}\big(C^2\big) + \frac{1}{2}\text{trace}\big(C\big)^2 $
to finish this off, via cyclic property of trace:
$\text{trace}\big(C\big) = \text{trace}\big(BA\big) = \text{trace}\big(AB\big)$
and
$\text{trace}\big(C^2\big) = \text{trace}\big((BA)^2\big) = \text{trace}\big(BABA\big)= \text{trace}\big(ABAB\big) = \text{trace}\big((AB)^2\big)$
so to complete the the problem you need to sum the diagonal elements of $AB$ and of $(AB)^2$.
Computational note: you there's no need to compute all of $(AB)^2$, you just need to compute it's 3 diagonal elements via 3 dot product operations, then sum them to get $\text{trace}\big((AB)^2\big)$.