I want to prove the following statement:
We define $SL(2,\mathbb{R})$ as the set of 2x2 matrix with determinat eiter 1 or -1.
G is a compact subgroup of $SL(2,\mathbb{R})$ if and only if $\exists C\in GL(2,\mathbb{R})$ s.t. $CMC^{-1}\in O(2,\mathbb{R})$ for all $M\in G$.
I think that the "if" part is easy:
If $\exists C\in GL(2,\mathbb{R})$ s.t. $CMC^{-1}\in O(2,\mathbb{R})$ for all $M\in G$ then G is contained in $C^{-1}(O(2,\mathbb{R}))C$ and since $O(2,\mathbb{R})$ is compact we can conclude.
Any suggestion for the "only if part"?
The proposition should be be: a closed subgroup (or a Lie subgroup) of $SL(2,R)$ is compact if and only if it is conjugated to a closed subgroup of $O(2,R)$ since you have subgroups of $O(2,R)$ which are not compact since the image of a compact set by a continuous map is compact.
Suppose that $MGM^{-1}\subset O(2,R)$ is closed, it is compact since it is a closed subset of a compact set. This implies that $G$ is compact since $f:SL(2,R)\rightarrow SL(2,R)$ defined by $f(X)=MXM^{-1}$ is a diffeomorphism, so $f^{-1}(f(G))=G$ is compact.
On the other hand, let $G$ be a compact subgroup of $SL(2,R)$, take the scalar product $b$ whose orthogonal group is $O(2,R)$ and define a new scalar product $b_G(x,y)=\int_Gb(g.x,gy)$ (there is a Haar measure on $G$), you obtain a scalar product $b_G$ invariant by $G$, this implies that $G$ is contained in the group $O(2,b)$ of linear matrices which preserve $b_G$, since $b$ and $b_G$ have the same signature, their orthogonal groups are conjugated (to see this take orthonormal bases for each bilinear forms and consider the base change matrix), so $G$ is conjugated to a subgroup of $O(2,R)$.