$$ A=\left[\begin{array}{lll} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{array}\right] $$ $$ D=\left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & -\left(\frac{1+\sqrt{3} i}{2}\right) & 0 \\ 0 & 0 & \left(\frac{-1+\sqrt{3} i}{2}\right) \end{array}\right] $$
$A$ is the form of the rotation matrix in $(x, y, z)$ basis (orthonormal) and $D$ is it's form in new basis $(x', y', z')$ in which it is diagonal. The rotation occurs about the axis in the direction$$\hat{n}=\frac{\hat{x}+\hat{y}+\hat{z}}{\sqrt{3}} $$
And the transformation rule between $(x, y, z)$ and $(x', y', z')$ is given as
\begin{aligned} x &=\frac{1}{\sqrt{3}}\left(x^{\prime}+\left(\frac{-1+\sqrt{3} i}{2}\right) y^{\prime}+\left(\frac{-1-\sqrt{3} i}{2}\right) z^{\prime}\right) \\ y &=\frac{1}{\sqrt{3}}\left(x^{\prime}+\left(\frac{\left.-1- \sqrt{3} i) y^{\prime}+(-1+\sqrt{3} i) z^{\prime}\right)}{2}\right)\right.\\ z &=\frac{1}{\sqrt{3}}\left(x^{\prime}+y^{\prime}+z^{\prime}\right) \end{aligned} So my question is how to find angle of rotation? Help!
As you know, your rotation is a rotaion around $\left(\frac1{\sqrt3},\frac1{\sqrt3},\frac1{\sqrt3}\right)$. Take a unitary vector $v$ orthogonal to this vector; for instance, take $v=\left(\frac1{\sqrt2},-\frac1{\sqrt2},0\right)$. It happens that$$A.v=\left(0,\frac1{\sqrt2},-\frac1{\sqrt2}\right)$$and so the angle $\theta$ that you are looking after is the angle between $v$ and $A.v$. Since these are two unit vectors, $\theta$ is such that $\cos\theta=\langle v,A.v\rangle=-\frac12$, and therefore $\theta=\frac{2\pi}3$.