This is a homework question I am unable to solve. Let $A,B \in PSL(2,R)=Aut(H)$. Assume none of them are elliptic and they have :
Case 1) one common fixed point at the boundary of $H$ (i.e. they could be either parabolic or hyperbolic),
Case 2) two common fixed points at the boundary of $H$ (i.e., both are hyperbolic).
Show that : $A,B$ are two elements of a Fuchsian group $\Gamma$ (discrete topological subgroup $\Gamma$ of $PSL(2,R)$) if and only if $A,B$ have a common power.
I will appreciate if you could kindly give me a detailed answer.
Also, there is one thing not clear to me, does the question mean: there is an integer $k$ so that $A^k=B^k$ or does it mean: there are integers $m,n$ so that $A^m=B^n$ ? A detailed answer will be appreciated! But if that is too much time consuming, could you give me some hints to solve this problem?
It means the latter: if to such integers exist, then there is a transformation which is a power of both of these, thus a common power. If there is such a common power, then $A$ and $B$ are in a certain (multiplicative instead of additive) sense commensurable.
If you choose $m$ and $n$ as small as possible with the above property, then you can show that “between” the identity and $A^m=B^n$ there are at most $mn$ distinct elements. As that number is finite, you have the discretenes you require for a Fuchsian group. Conversely you can show that if there is no common power, then the group generated by $A$ and $B$ must contain infinitely many group elements in any segment.
As you migt have noticed, I have used an intition about “between-ness” in my above formulation. For that I imagine the group generated by $A$ and $B$. If $A$ and $B$ have all their fixed points in common, then so will all the other elements of that group, and you will wither have translations along a common geodesic (hyperbolic case) or limit rotations around a common ideal point (parabolic case). In both cases, a concept of “between” is easy to understand from intuition, and shouldn't be much harder to formulate rigurously.
If $A$ and $B$ do not have all their fixed points in common, then either they are two translations sharing one and only one fixed point, or one is a limit rotation and one translation. Both of these cases cannot happen for Fuchsian groups, and both of these will never have a common power. So even in this case, the “if and only if” condition you try to prove still holds.