I derived some stuff and it is happening that i come to the following expression:
$\frac{r_{13}^2 + r_{23}^2}{(r_{11}r_{23} - r_{13}r_{21})^2 + (r_{12}r_{23} - r_{13}r_{22})^2}$
that must equal 1 for all first two rows of a rotation matrix (orthogonal matrix).
$$ \begin{matrix} r_{11} & r_{12} & r_{13} & \\ \end{matrix} $$ $$ \begin{matrix} r_{21} & r_{22} & r_{23} & \\ \end{matrix} $$
I am confused how to proof this! I see a bit of a cross product in the denominator but can't identify the relation combined with the numerator.
I did this just by using the facts that the rows and columns have length one and are mutually orthogonal.
$$\begin{align} &(r_{11}r_{23}-r_{13}r_{21})^2+(r_{12}r_{23}-r_{13}r_{22})^2\\ &=r_{11}^2r_{23}^2+r_{13}^2r_{21}^2-2r_{11}r_{21}r_{13}r_{23}+ r_{12}^2r_{23}^2+r_{13}^2r_{22}^2- 2r_{12}r_{22}r_{13}r_{23}\\ &=r_{23}^2(r_{11}^2+r_{12}^2)+r_{13}^2(r_{21}^2+r_{22}^2) -2r_{13}r_{23}(r_{11}r_{21}+r_{12}r_{22})\\ &=r_{23}^2(1-r_{13}^2)+r_{13}^2(1-r_{23}^2)-2r_{13}r_{23}(-r_{13}r_{23})\\ &=r_{13}^2+r_{23}^2-2r_{13}^2r_{23}^2+2r_{13}^2r_{23}^2\\ &=r_{23}^2+r_{13}^2 \end{align}$$
I see that an answer has been posted since I started typing this (I'm awfully slow), but I'm going to post it anyway, since it's a bit different.