If conjugation by a matrix preserves the matrix norm then the matrix must be conformal?

Question

Let $A$ be an $n \times n$ real invertible matrix, and suppose that $\| X\|^2=\| AXA^{-1}\|^2$ for every $n \times n$ real matrix $X$.

Is it true that $A$ must be conformal?

(It is easy to see that conformal matrices $A$ satisfy this).