if $A$ is an $m\times n$ matrix and $B$ is an $n\times m$ matrix prove that
$$\det \left( \begin{bmatrix} O &A \\-B & I \end{bmatrix} \right)=\det(AB)$$
tried many thing such as getting it to triangle form but I could not maybe you can, or can give alternative answer
You can get it into triangular form by means of column operations. Or look at it this way:
$$ \begin{pmatrix} O & A\\ -B & I \end{pmatrix} \begin{pmatrix} I & O\\ B & I \end{pmatrix} = \begin{pmatrix} AB & A\\ O & I \end{pmatrix} $$ and taking determinants gives the result.