I am stuck on the following two questions. I find formulas for the computation of 3D rotation matrix, but still cannot get how to do those questions.
Find matrix for rotation $R_{\theta \bar n}$, where $\bar n=\frac{1}{\sqrt 3}(\bar i+\bar j+\bar k)$ and $\theta=\pi/2$
Let $R$ be rotation, given by $R(\bar i)=\bar j, R(\bar j)=\bar k, R(\bar k)=\bar i$. Find corresponding matrix and corresponding vector for the rotation.
I will not write your matrices for you, but the following should be a big hint about how you can write them.
Consider this linear transformation matrix: $$ A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix}. $$
And consider the usual representation of $\bar i$ as a numeric vector: $$ \bar i = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}. $$
Apply the linear transformation described by the matrix $A$: $$ A\, \bar i = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix} \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 4 \\ 7 \end{pmatrix}. $$
Now suppose you have some linear transformation with matrix $B$, and you find out that $$ B\,\bar i = \begin{pmatrix} 1 \\ 4 \\ 7 \end{pmatrix}. $$
What does that tell you about the first column of $B$?
Of course the specific matrices $A$ and $B$ described above are not rotation matrices, but the ideas above work for any linear transformation matrix $M$, and rotations are just a particular kind of linear transformation.
Part (2) of your question should be easy now. Part (1) can also be solved using this hint, but first you have to figure out what happens to each of the vectors $\bar i$, $\bar j$, and $\bar k$ when you rotate them by an angle $\pi/2$ around the vector $\frac{1}{\sqrt 3}(\bar i+\bar j+\bar k)$.