$\mbox{tr}((AB)^2) = \mbox{tr} (A^2B^2)$

Question

Let $\mathbb K$ be a field with characteristic $3$ and $A, B \in M_3(\mathbb K)$ such that $\mbox{tr}((AB)^2) = \mbox{tr} (A^2B^2)$.

1. Show that $(AB-BA)^3$ is a scalar matrix.

2. Find a couple solution $(A,B) \in M_3 (\mathbb F_3)$ such that $AB-BA$ is not a scalar matrix.

Attemp: I tried to develop the cube and that of properties of trace.

Answer 1

Note that (using the properties of trace), $$ \operatorname{tr}[(AB - BA)^2] = \operatorname{tr}(ABAB - ABBA - BAAB + BABA) \\ = 2\operatorname{tr}((AB)^2) - 2\operatorname{tr}(A^2B^2). $$ So, the $M = AB - BA$ satisfies $\operatorname{tr}(M) = 0$ and $\operatorname{tr}(M^2) = 0$. Now, the characteristic polynomial of $M$ is given by $$ \det(M - tI) = -t^3 + \operatorname{tr}(M)t^2 - \frac 12 [\operatorname{tr}^2(M) - \operatorname{tr}(M^2)] t+ \det(M) \\= -t^3 + \det(M). $$ By the Cayley-Hamilton theorem, it follows that $$ 0 = -M^3 + \det(M) I \implies M^3 = \det(M) I. $$ So indeed, $M^3$ is a scalar matrix.

---

For part 2, consider $$ A = \pmatrix{0&1&0\\0&0&0\\0&0&0}, \quad B = \pmatrix{0&0&0\\1&0&0\\0&0&0}. $$