Trivial elements in $T(a,b,c)$

Question

Consider the group $T(a,b,2)=<x,y|x^a, y^b, (xy)^2>$ and assume none of $a$ or $b$ is equal to $2$. How can one list all the trivial words (say up to length $11$ and apart from $(xy)^{2n})$) in this group.

Answer 1

In the case that $\frac{1}{a}+\frac{1}{b} + \frac{1}{c} < 1$, which for your situation $c=2$ amounts to $\frac{1}{a} + \frac{1}{b} < \frac{1}{2}$, Dehn's Algorithm is a computational algorithm for checking whether a word represents the identity element in this groups.

To justify Dehn's algorithm we need two facts.

First, $T(a,b,2)$ is a word hyperbolic group when the above inequality holds; that's a consequence of applying the Poincare polygon theorem to a hyperbolic polygon with angles $\pi/a$, $\pi/b$, $\pi/2$, reflecting that polygon across a side and taking the union to get a fundamental domain for an action of $T(a,b,2)$ on the hyperbolic plane.

Second, every word hyperbolic group has some finite set $R$ of defining relations that satisfies Dehn's algorithm; see the book by Epstein, Cannon, Holt, Levy, Paterson, Thurston.

So the conclusion is that Dehn's algorithm for some defining relators $R$ applies to your group $T(a,b,2)$ with respect to some presentation, and this is what allows you to list out all trivial words. Now, I am a little unsure whether your exact presentation is a Dehn's algorithm presentation for $T(a,b,2)$ when the inequality above holds; although I am 95% sure that it is and I think you can verify this by carrying out Dehn's original proof of his algorithm for the usual presentations of surface groups.

The statement of Dehn's algorithm, and the answer to your question assuming you have the correct defining relators $R$, is as follows. For any word $w$ in $x,y,z$ and their inverses, the word $w$ represents the identity element of the group if and only if iteration of the following process eventually reduces $w$ to the empty word: if $r \in R$ is one of the defining relators or their inverses or the cyclic permutations thereof, if $r=uv$ where $Length(u) > \frac{1}{2} Length(r)$, and if $w$ has a subword $u$, rewrite $w$ by replacing the $u$ subword with $v$.