behavior of scalar product defined by trace under commutator

Question

Define $$\langle X,Y \rangle := \operatorname{tr}XY^t,$$ where $X,Y$ are square matrices with real entries and $t$ denotes transpose.

I have some troubles in proving that $$ \langle [X,Y],Z \rangle = - \langle Y,[X,Z] \rangle,$$ where square brackets denote commutator.

Let me update my questin to part ii. You have proven that my commutation relation without a transpose is wrong, while it is correct if we put a $t.$

Then I'd say I'm in trouble, because the next step would be to define $$\operatorname{ad}_XY:=[X,Y]$$ and claim that by the above (false) property we have that $\operatorname{ad}$ is antisymmetric, i.e. $$\langle \operatorname{ad}_XY,Z\rangle =- \langle Y,\operatorname{ad}_XZ\rangle:$$ Do you know of a way to recover such a nice property or something similar?

Answer 1

The key properties to use are

$$\langle A,B\rangle=\langle B, A\rangle,$$

i.e. with $tr((AB)^t)=tr(AB),$ and

$$tr(ABC^t)=tr(BC^tA),$$

for all $A,B\in M_n(\mathbb R)$.

Answer 2

$$\langle XY-YX,Z\rangle=\langle XY,Z\rangle-\langle YX,Z\rangle\\=\langle Y,X^tZ\rangle-\langle Y,ZX^t\rangle\\=\langle Y,[X^t,Z]\rangle$$ Counterexample for the OP's equation: Let $X=Z=\left[\array{0 & 1\\0 &0}\right]$ and $Y=\left[\array{1&0\\0&2}\right]$. We have $XZ-ZX=0\implies \langle Y,[X,Z]\rangle=0$, whereas $$X^tZ-ZX^t=\left[\array{-1&0\\0&1}\right]\implies\langle Y,[X^t,Z]\rangle=1.$$