I wish to find the normal stress at the point on an interface whose normal unit vector is
$$n= \begin{bmatrix} 0.6 & 0.8 & 0\\ \end{bmatrix}$$
given the stress tensor $$\tau_{ij}= \begin{bmatrix} 2000 & 500 & -1000\\ 500 & 3000 & 3000\\ -1000 & 3000 & -4000\\ \end{bmatrix}$$
my goal is to rotate the stress tensor so the rotated $x$ axis ($x'$) is coalesces with the normal vector $n$ such that the new rotated $\tau_{x'x'}$ should be the answer.
We have that
$$\alpha=\arctan\left(\frac{0.8}{0.6}\right)=53.13 $$ using the rotatin matrix about the $Z$ axis $$R_z=\begin{bmatrix} \cos(53.13) & -\sin(53.13)& 0\\ \sin(53.13) & \cos(53.13)& 0\\ 0& 0 & 1\\ \end{bmatrix}$$
the rotated tensor is: $$\tau_{i'j'}=R\tau_{ij}R^T=\\ \begin{bmatrix} \cos(53.13) & -\sin(53.13)& 0\\ \sin(53.13) & \cos(53.13)& 0\\ 0& 0 & 1\\ \end{bmatrix} \begin{bmatrix} 2000 & 500 & -1000\\ 500 & 3000 & 3000\\ -1000 & 3000 & -4000\\ \end{bmatrix} \begin{bmatrix} \cos(53.13) & \sin(53.13)& 0\\ -\sin(53.13) & \cos(53.13)& 0\\ 0& 0 & 1\\ \end{bmatrix}\\ =\begin{bmatrix} 2160 & -620 & -3000\\ -620 & 2840 & 1000\\ -3000 & 1000 & -4000\\ \end{bmatrix}$$
which yields a wrong answer: $\tau_{x'x'}=2160$.
But $\tau_{\textbf{i'j'}}=R^T\tau_{ij}R$ yields the correct answer:
$$\tau_{\textbf{i'j'}}=\\ \begin{bmatrix} \cos(53.13) & \sin(53.13)& 0\\ -\sin(53.13) & \cos(53.13)& 0\\ 0& 0 & 1\\ \end{bmatrix} \begin{bmatrix} 2000 & 500 & -1000\\ 500 & 3000 & 3000\\ -1000 & 3000 & -4000\\ \end{bmatrix} \begin{bmatrix} \cos(53.13) & -\sin(53.13)& 0\\ \sin(53.13) & \cos(53.13)& 0\\ 0& 0 & 1\\ \end{bmatrix}\\ =\begin{bmatrix} 3120 & 340 & 1800\\ 340 & 1880 & 2600\\ 1800 & 2600 & -4000\\ \end{bmatrix}$$
$\tau_{\textbf{i'j'}}=\tau_{x'x'}=3120$.
Why is that?
Rotation matrix $R_z$ represents the change from the new basis $\mathbb B$' to the old one $\mathbb B$, therefore
$$R_z\,T'=\tau \; R_z\,n' \implies T'=R_z^T\,\tau \, R_z\,n'$$
then in the new basis $\tau'=R_z^T\,\tau \, R_z$.
Refer also to
The key point is that matrix $R_z$ represents the change from the new basis $\mathbb B$' to the old one $\mathbb B$, indeed in the new basis the normal vector is $n'=[1\;\;0\;\;0]$ and then
$$R_z\,n' = n=[0.60\;\; 0.80\;\;0]$$
which is indeed the normal vector in the old basis.