I'm trying to find two rotation matrices in $\mathbb R^3$, one about the line spanned by $(1,1,0)$ through $\pi/2$, and the other one about the line spanned by $(1,1,1)$ through $2\pi/3$. Now, some algorithms of doing the first problem have already been discussed here (see e.g. this), but I fail to implement them.
I found that $T_1$ suggested in @Jyrki Lahtonen's answer is
\begin{pmatrix} \frac{1}{\sqrt 2}&-\frac{1}{\sqrt 2} & 0 \\ \frac{1}{\sqrt 2}&\frac{1}{\sqrt 2} & 0 \\ 0& 0 & 1 \end{pmatrix}
$T_2$ is
\begin{pmatrix} 1&0 & 0 \\ 0&0 & -1 \\ 0& 1 & 0 \end{pmatrix} and $T_1^{-1}=T_1^t$. But then $T_1T_2T_1^{-1}$ is \begin{pmatrix} 1/2&1/2 & 1/\sqrt{2} \\ 1/2&1/2 & -1/\sqrt{2} \\ -1/\sqrt{2}& 1/\sqrt{2} & 0 \end{pmatrix} which is not even orthogonal. What am I doing wrong?
For the second matrix, I followed the second answer in the above-mentioned question. Let $v_1=\frac{1}{\sqrt 3}(1,1,1)^t, v_2=\frac{1}{\sqrt{2}}(1,-1,0)^t, v_3=\frac{v_1\times v_2}{|v_1\times v_2|}=\frac{1}{\sqrt 6}(1,1-2)$. Let $E=(v_1 | v_2 | v_3)$ and \begin{equation} M' = \begin{pmatrix} 1&0&0\\ 0&\cos{\theta} & -\sin{\theta} \\ 0&\sin{\theta} & \cos{\theta} \end{pmatrix} \end{equation} where $\theta=2\pi/3$. Then the matrix I need is $M=EM'E^{-1}$. Is that correct?
$M = EM'E^{-1}$
$E^{-1}$ Takes a vector in the standard basis to a basis such that the axis of rotation maps to $v_1$
$M'$ is a rotation of $\theta$ about the axis of $v_1$
$E$ transforms back to the standard basis.
However, to visualize 1/3 revolution about the axis (1,1,1). Imagine a cube. Three of the edges lie on the coordinate axes. The axis of rotation runs through the middle of the cube. 1/3 revolution, maps the cube to itself.
$(1,0,0)$ maps to $(0,1,0)$
$(0,1,0)$ maps to $(0,0,1)$
$(0,0,1)$ maps to $(1,0,0)$
$\begin{bmatrix} 0&0&1\\1&0&0\\0&1&0\end{bmatrix}$