The group:
$$ G = \left\langle x, y \; \left| \; x^2 = y^3 = (xy)^7 = 1\right. \right\rangle $$
is infinite, or so I've been told. How would I go about proving this? (To prove finiteness of a finitely presented group, I could do a coset enumeration, but I don't see how this helps if I want to prove that it's infinite.)
On
Grigory has already answered your particular question. However, I wanted to point out that your question "How do you prove that a group specified by a presentation is infinite?" has no good answer in general. Indeed, in general the question of whether a group presentation defines the trivial group is undecidable.
On
Your group is the alternating subgroup of a Coxeter group
$$\langle x, y, z | x^2 = y^2 = z^2 = (xy)^2 = (yz)^3 = (zx)^7 = 1 \rangle$$
which is not on the (well-understood) list of finite Coxeter groups, and the alternating subgroup always has index $2$. (The connection to the hyperbolic plane is that Coxeter groups of rank $3$ always act as symmetries of a tiling of either the sphere, the Euclidean plane, or the hyperbolic plane by triangles.)
On
Another approach, which can't work in general (see Noah's answer), but will surely work in this case, is to find a normal form for each element of the group, and then see whether there are finitely or infinitely many.
In practice, that means imagining a word in x and y, and then applying the relations as much as possible to simplify it, and then trying to figure out (and prove!) what the possible different "irreducible words" (i.e. words that can no longer be simplified) are.
In your case, the first thing one would note is that x can only appear to the 1st power (since any higher power can be simplified using x^2 = 1), while y can only appear to the powers +1 or -1 (for the same reason). Also, we can't have too many expressions of the form xy or yx in a row, because of the third relation.
One can keep going like this. I didn't, but what I imagine is that one can have expressions of the form x y x y^{-1} x y x y^{-1} ... that are arbitrarily long, and inequivalent, explaining the infinite order of the group.
It shouldn't be so hard to settle the question from this point of view by sitting down with pencil and paper and just playing around with different words to get a feel for what kinds of reductions can take place. (In geometric arguments like Grigory's, one uses a geometric context, such as an action on a hyperbolic tiling, as a more conceptual way of understanding the relations and distinguishing inequivalent words. But in this case I'm sure it won't be hard to see everything directly from the presentation.)
Added after rereading the question: what I am suggesting is precisely that even when the answer might be infinite, you can still hope to find a coset enumeration, at least in a case like this with relatively simple relations.
On
The group:
$$G = \left\langle x, y \mid x^l = y^m = (xy)^n = 1 \right\rangle$$ is triangular group so, if $\frac{1}{l} + \frac{1}{m} + \frac{1}{n} \lt 1$, then this group is infinite...
On
One general way to do this, (which is not guaranteed to work, as Noah points out), is to exhibit infinitely many different homomorphic images of the group you start with . In this case, any group generated by an element of order $2$ and an element of order $3$ whose product has order $7$ is a homomorphic image of the group $G$. Such a group is a Hurwitz group, and these are well studied, for example in the work of M. Conder and of G. Higman, among others. Infinitely many finite simple groups are known to be Hurwitz groups, I believe.
$\langle x,y \; | \; x^2=y^3=1 \rangle \cong \operatorname{PSL}_2(\mathbb Z)$ and this isomorphism identifies G with $\operatorname{PSL}_2/T^7=1$ (where $T:z\mapsto z+1$). Result is the symmetry group of the tiling of the hyperbolic plane. From this description one can see that G is infinite (e.g. because there are infinitely many triangles in the tiling and G acts on them transitively).