If $\lambda$ and $\mu$ are the (complex) eigenvalues of $A$, then $\lambda\mu=\det(A)=1$, hence
$$\mathrm{tr}(A^2)=\lambda^2+\mu^2=(\lambda+\mu)^2-2\lambda\mu=\mathrm{tr}(A)^2-2\geq-2$$
Note that $\mathrm{tr}(A)$ is real even if $\lambda$ and $\mu$ are not.
If $\lambda$ and $\mu$ are the (complex) eigenvalues of $A$, then $\lambda\mu=\det(A)=1$, hence $$\mathrm{tr}(A^2)=\lambda^2+\mu^2=(\lambda+\mu)^2-2\lambda\mu=\mathrm{tr}(A)^2-2\geq-2$$ Note that $\mathrm{tr}(A)$ is real even if $\lambda$ and $\mu$ are not.