I recently took an algebra test, and one of the questions was like this (apologise upfront because I don't remember the exact matrix elements, but the idea is essentially the same):
Let $A$ and $B$ be two square matrices. We know that: $$AB=\begin{bmatrix}1 & 2 \\ 3&4\end{bmatrix}, \quad BA=\begin{bmatrix}3 & a \\b &2\end{bmatrix}$$ What can be said about values $a$ and $b$? What possible values can they have?
The real exam problem gave me only 4 options and only 1 was correct (it was a multiple-choice test question). The first 3 of the answers suggested possible values for $a$ and $b$, whereas the last answer stated $\text{(d)Nothing can be said about a and b}$.
How would you proceed? I spent like 10 minutes trying several things to no avail. No further conditions were given on A nor B. We only know they are square and 2x2.
What I tried:
I stablished: $$AB=X, \quad BA=Y$$ Then I multiplied by the inverse of $B$ (assuming it has an inverse), to reach: $$A=XB^{-1}=B^{-1}Y$$ $$B=A^{-1}X=YA^{-1}$$ But I really don't know where this will lead me.
Nothing can be said besides the fact that $a\neq0$ and that $b=\frac 8a$ (which means that both matrices $\left(\begin{smallmatrix}1&2\\3&4\end{smallmatrix}\right)$ and $\left(\begin{smallmatrix}3&a\\b&2\end{smallmatrix}\right)$ have the same determinant). Indeed, if you take$$A=\begin{pmatrix}0&\frac a4\\1&\frac a8\end{pmatrix}\text{ and }B=\begin{pmatrix}\frac52&3\\\frac4a&\frac8a\end{pmatrix},$$then$$AB=\begin{pmatrix}1&2\\3&4\end{pmatrix}\text{ and }BA=\begin{pmatrix}3&a\\\frac8a&2\end{pmatrix}.$$