How to prove this matrix inequality？

Question

Given the following two conditions:

(1). $A$ and $B$ are $n\times n$ matrices;

(2). $U$ is a unitary $n\times n$ matrix, i.e., $UU^{\ast}=U^{\ast}U=1$

How to prove this matrix inequality: tr$\left( AUBU^{\ast}\right) \leq $tr$\left( AB\right) $ ?

Answer 1

Here is a proof for real matrices. I don't think it makes sense to prove this for complex matrices as trace can then be complex valued and they cannot be ordered.

By Spectral Theorem for normal matrices, we get that for unitary matrix $U$ is diagonalizable and in particular all of its eigenvalues have radius 1. For the proof, observe the following: let $U$ be unitary and $Uv=\lambda v$. Then,

$$||Uv||^2 = \langle Uv, Uv \rangle = \langle v, U^*Uv\rangle= ||v||^2$$ Also, $$||Uv||^2 = |\lambda|^2 ||v||^2$$

Hence,$ |\lambda| = 1$ or $\lambda=-1,1$.

Then, using properties of trace,

$$tr(AUBU^*) \leq tr(AUB)tr(U^*) = tr(UBA)tr(U^*) \leq tr(U)tr(U^*)tr(BA) = tr(AB)$$

where the last equality is from the observation that trace of a matrix is the sum of eigenvalues and that for unitary matrix $U$ and $U^*$, the eigenvalues are again $-1,1$ with same multiplicities.

EDIT: The last assertion still needs work and may need more assumptions.

Answer 2

This is only true for Hermitian matrices and does not need to hold otherwise. For example, if $$ A=B=\begin{bmatrix}1 & 1\\0 & 1\end{bmatrix}, \quad U=\begin{bmatrix}0 & 1\\ 1 & 0\end{bmatrix}, $$ then $$ \mathrm{trace}(AUBU^*)=3\not\leq 2=\mathrm{trace}(AB). $$