I know how to find the linear transformation with angle rotation, but I don't know how it can be applied to a vector only in a certain axis.
For example: rotation of $v = [5, 2, 3]^T$ about the z-axis through the angle $5pi/6$
I can find the linear transformation of the z-axis:
cos 150 -sin 150
sin 150 cos 150
But I don't know how to apply it to the vector v when it's rotating about the z-axis.
Please help.
The matrix $M$ of the transformation is $3\times 3$. We can obtain this matrix by considering the image of $\displaystyle \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$, $\displaystyle \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}$ and $\displaystyle \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}$.
The image of $\displaystyle \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$ is $\displaystyle \begin{pmatrix} \cos\theta \\ \sin \theta \\ 0 \end{pmatrix}$. This is the first column of $M$.
The image of $\displaystyle \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}$ is $\displaystyle \begin{pmatrix} \cos(\frac{\pi}{2}+\theta) \\ \sin (\frac{\pi}{2}+\theta) \\ 0 \end{pmatrix}=\begin{pmatrix} -\sin\theta \\ \cos \theta \\ 0 \end{pmatrix}$. This is the second column of $M$.
The image of $\displaystyle \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}$ is $\displaystyle \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}$. This is the third column of $M$.
$$M=\begin{pmatrix} \cos\theta & -\sin\theta &0 \\ \sin\theta & \cos\theta &0 \\ 0&0&1 \end{pmatrix}$$
The image of $v$ is
$$\begin{pmatrix} \cos150^\circ & -\sin150^\circ &0 \\ \sin150^\circ & \cos150^\circ &0 \\ 0&0&1 \end{pmatrix}\begin{pmatrix} 5 \\ 2 \\ 3 \end{pmatrix}=\frac{1}{2}\begin{pmatrix} -\sqrt{3} & -1 &0 \\ 1 & -\sqrt{3} &0 \\ 0&0&1 \end{pmatrix}\begin{pmatrix} 5 \\ 2 \\ 3 \end{pmatrix}=\frac{1}{2}\begin{pmatrix} -5\sqrt{3}-2 \\ 5-2\sqrt{3} \\ 3 \end{pmatrix}$$