Question: A rotation through $45^{\circ}$ about the x-axis is followed by a similar one about the z-axis. Show that the rotation corresponding to their combined effect has its axis inclined at equal angles $cos^{-1}\dfrac{1}{\sqrt{5-2\sqrt{2}}}$ to the x and z axes.
I tried By Rodrigues' Rotation Theorem - or just by the standard rotation matrices - the first rotation, through $45^{\circ}$ about the x-axis, can be described by $R_1 = \left[ \begin{matrix} 1 & 0 & 0 \\ 0 & \tfrac{1}{\sqrt{2}} & \tfrac{-1}{\sqrt{2}} \\ 0 & \tfrac{1}{\sqrt{2}} & \tfrac{1}{\sqrt{2}} \\ \end{matrix} \right]$.
The second rotation, through $45^{\circ}$ about the z-axis, can be described by $R_2 = \left[ \begin{matrix} \tfrac{1}{\sqrt{2}} & \tfrac{-1}{\sqrt{2}} & 0 \\ \tfrac{1}{\sqrt{2}} & \tfrac{1}{\sqrt{2}}& 0 \\ 0 & 0 & 1 \\ \end{matrix} \right]$. So the combined rotation is $ R_2R_1 = = \left[ \begin{matrix} \tfrac{1}{\sqrt{2}} & \tfrac{-1}{2} & \tfrac{1}{2} \\ \tfrac{1}{\sqrt{2}} & \tfrac{1}{2}& \tfrac{-1}{2} \\ 0 & \tfrac{1}{2} & \tfrac{1}{2} \\ \end{matrix} \right]$. At last - by Rodigues' Rotation Theorem - $\cos\theta = \dfrac{1}{2}\left(Trace(R_2R_1) - 1\right) = \frac{1}{2}\left(\sqrt{2} + \frac{1}{2} - 1 \right)$ --- which isn't the right answer. What went wrong? Thank you.
Your first mistake is in the third row of $R_2 R_1$. When you have that, a vector $\bf v$ in the direction of the axis is an eigenvector of $R_2 R_1$ for eigenvalue $1$. The cosine of the angle between this and the unit vector $\bf u$ is ${\bf u} \cdot {\bf v}/\sqrt{{\bf v} \cdot {\bf v}}$.