$n\geq3$. A and B are two $n\times n$ reals matrices. For $n\times n$, Could one give counterexamples to show that
$$ \det \Big(A^2+B^2\Big)\ge \det(AB-BA) \tag{$*$}$$
is not necessarily true?
Well, I won't do more than $3\times3$: the following $A,B$ show (*) is not right
$$A=\left(\begin{array}{ccc}0&2&0\\0&0&1\\0&0&0\end{array}\right), B=\left(\begin{array}{ccc}0&0&0\\1&0&0\\0&1&0\end{array}\right)$$
What is the counterexample for $n\times n$? Thanks a lot!
Let \begin{equation}A=\begin{bmatrix}I_{n-2} & \mathbf{0} & \mathbf{0} \\ \mathbf{0} & 0 & -1 \\ \mathbf{0} & 1 & 0\end{bmatrix},\end{equation} then \begin{equation}A^2=\begin{bmatrix} I_{n-2} & \mathbf{0} & \mathbf{0} \\\mathbf{0}&-1 & 0 \\ \mathbf{0}& 0 & -1\end{bmatrix}.\end{equation} Now make $B$ a matrix filled with zeros, except for the very last diagonal entry, make this entry $\sqrt{2}$. Then $B^2$ has all zeros except for the last entry on the diagonal which is $2$, and so we have \begin{equation}A^2+B^2=\begin{bmatrix} I_{n-2} & \mathbf{0} & \mathbf{0} \\\mathbf{0}&-1 & 0 \\ \mathbf{0}& 0 & 1\end{bmatrix}.\end{equation} We then also have \begin{equation}AB-BA=\begin{bmatrix} \mathbf{0} & \mathbf{0} & \mathbf{0} \\\mathbf{0}&0 & -\sqrt{2} \\ \mathbf{0}& -\sqrt{2} & 0\end{bmatrix},\end{equation} and so then det$(A^2+B^2)=-1$ and det$(AB-BA)=0$ for all $n \geq 3$.