Let $M$ be a real $2\times2$ matrix $$ M = \left(\begin{matrix}a & b \\ c & d\end{matrix}\right) $$ Suppose $M$ has a finite order (thus there is some natural $n$ that $M^n=E$, where $E$ is identity matrix), is it necessary a rotational matrix $R_\varphi$ where $\varphi = \frac{m}k\pi$?
First of all I noticed, that $\det(M)=\pm1$. Consider now only $+1$, thus $ad-bc=1$. I thought that if there is at least one real eigenvalue $\lambda$, then there is at least one real eigenvector, then the whole plane stretches to that direction. It means that constant multiplication by $M$ would not reverse this stretching. Thus eigenvalues must be complex: $$ \lambda^2 - (a+d)\lambda + ad-bc = 0 $$ $$ \lambda^2 - (a+d)\lambda + 1 = 0 $$ $$ \lambda \notin \mathbb{R} \Leftrightarrow|a+d| < 2 $$ Rotational matrix $R_\varphi$ has $a = d$, but the condition above doesn't have this restriction. So it is not hard to construct a matrix with determinant 1 and with $a\ne d$, for example $$ M = \left(\begin{matrix}\frac34 & \frac12 \\ -\frac12 & 1\end{matrix}\right) $$ Does this matrix has a finite order too? I think not. What am I missing? Is it true, that finite order $\Rightarrow$ rotation on the rational fraction of $\pi$? How to derive this fact?
You made a good start, but then went a bit astray when you concluded that if $\det(M)=1$, then its eigenvalues must be complex. This is clearly not true, though: $E^n=E$ for all $n$, and its only eigenvalue is $1$.
$M$ having finite order $n$ means that the polynomial $\lambda^n-1$ annihilates $M$. Since the minimal polynomial of $M$ divides this polynomial, this in turn implies that the only possible eigenvalues of $M$ are $n$th roots of unity. The characteristic polynomial of $M$ is quadratic, so its eigenvalues are either both real or both complex.
There are three cases to consider for real eigenvalues: repeated $1$, repeated $-1$ and $\{1,-1\}$. (Note that we can only have $-1$ as an eigenvalue if $n$ is even.) In the first case, $M=E$ and in the second, $M=-E$ (why?), which can be interpreted as a rotation through an angle of $\pi$. If we have both $1$ and $-1$ as eigenvalues, then $M^2=E$ (why?), which is one of the hallmarks of a reflection. Looking at it geometrically, $M$ reverses one direction and leaves another fixed. Note that this is the only case in which $\det(M)=-1$, so if you require that $\det(M)=1$, then you can exclude reflections.
Unfortunately, even this additional hypothesis doesn’t get you pure rotations only. If $M$ has complex eigenvalues, they must be a complex conjugate pair $e^{\pm i\phi}$, and their product is $1$. In this case, $M$ is similar to (but not necessarily equal to!) a matrix of the form $$R_\phi = \pmatrix{\cos\phi&-\sin\phi\\\sin\phi&\cos\phi}.$$ Why not equal? Well, if $R_\phi^n=E$ and $P$ is any invertible $2\times2$ matrix, then $\left(PR_\phi P^{-1}\right)^n = PR_\phi^nP^{-1} = E$ as well. Matrices of the form $PR_\phi P^{-1}$ are known as conjugate rotations. Geometrically, $R_\phi\mathbf v$ traces out a circle as $\phi$ varies (with $\mathbf v\ne0$, of course), while $PR_\phi P^{-1}\mathbf v$ traces an ellipse.