Consider a fixed infinite group, such as $SL(n, \mathbb{R})$. Let $S$ be a given finite subset of the elements of $SL(n, \mathbb{R})$, further suppose that $S$ is closed under inverses. Is there any way to decide whether $S$ generates a finite or infinite subgroup of $SL(n, \mathbb{R})$?
It would seem that choosing some random finite set of elements in $SL(n, \mathbb{R})$ (and their inverses), it should be far more likely that the set generates an infinite subgroup rather than a finite one. However, I am not sure how to begin trying to decide a priori whether or not a given subset generates a finite or infinite subgroup of the original group. My choice of $SL(n, \mathbb{R})$ was largely arbitrary; does this problem become any easier if instead we choose $SL(n, \mathbb{Z})$ or some other convenient matrix group?
The problem with computing over the real numbers is that calculations can only be done to some fixed precision, so the accuracy of the answer is unclear.
But the answer to your question is yes for finitely generated subgroups $G$ of ${\rm GL}(n,K)$ where $K$ is a field in which you can do exact computation, such as a number field or a function field.
This algorithm is implemented in Magma and it is is amazingly fast and, in the case when $G$ is finite, an isomorphic image of $G$ in ${\rm SL}(n,F)$ for some finite field $F$ is also returned.
This algorithm is described in the paper
Detinko, A.S., Flannery, D.L., O’Brien, E.A., 2013. Recognizing finite matrix groups over infinite fields. J. Symb. Comput. 50, 100–109.