I have the next problem:
Find the values for wich the matrix is a rotation ($M\cdot M^{t}=Id$, $\det M= 1$): \begin{equation*} M= \left( \begin{array}{ccc} a & * & 0 \\ b & * & * \\ c & * & * \end{array} \right) \; \; \; i f \; \; \; bc\neq 0 \end{equation*} where $a, b, c$ are known constants.
I tried to apply the two conditions but I got stuck. This is my proccedure:
\begin{align*} M\cdot M^{t}&= \left( \begin{array}{ccc} a & x_1 & 0 \\ b & x_2 & x_4 \\ c & x_3 & x_5 \end{array} \right) \left( \begin{array}{ccc} a & b & c \\ x_1 & x_2 & x_3 \\ 0 & x_4 & x_5 \end{array} \right) \newline &= \left( \begin{array}{ccc} a^2 +x_1^2 & ab + x_1 x_2 & ac +x_1 x_3 \\ ab+ x_1 x_2&b^2+x_2^2+x_4^2& bc+x_2x_3+x_4 x_5 \\ ac+x_1 x_3 & bc+x_2 x_3 + x_4 x_5 & c^2 + x_3^2 +x_5^2 \end{array} \right) \newline &= \left(\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right) \end{align*}
That give me 6 equations and with the condition $\det M =1$ I get $a(x_2 x_5 - x_4 x_3)-x_1(bx_5 -cx_4)=1$ and I don't know what to do now. I would appreciate any help.
If $M^{T}M = \mathcal{I}$ then the columns vectors of $M$ are orthonormal to each other (orthonormal because the determinant is unity: $\textrm{det}(M^{T}M) = \textrm{det}(M)^2=\textrm{det}(\mathcal{I})=1$): $$m_i^{T}m_k=\delta_{ik}$$ being $m_i$ the $i$-th column of $M$.
From this you have (supposing $|(a,b,c)|=1$): $$(0, m_{23}, m_{33})^{T}(a, b, c)=0$$ giving $m_{23}=c/\sqrt{b^2+c^2}$ and $m_{33}=-b/\sqrt{b^2+c^2}$ (for example)
The last vector must be perpendicular to the other two and be part of a right handed basis to preserve orientation ($\textrm{det}M=1$), therefore: $m_2=m_3\times m_1/|m_3\times m_1|$