I'm studying group characters and representation on Hill's book, for fun self study. One of the exercises ask to verify the following presentation contains contradictory relations: $$\langle x, y : x^3 = y^3 = 1, yx = x^2y \rangle$$ The first concern I have is the book is claiming "a group of order 9 would presumably result", and it's not fully clear how it can estimate that.
The second is how to find a solution. I mean, the book also suggests that $x^3=1$ implies $x^2\neq1$ and $x\neq1$, so I thinking what really means to establish a contradiction.
If I was able to confirm the expected order is 9, I could try to build some table and verify the generated elements would be more than that value, but it would be eventually too mechanic and not sure if feasible neither.
For example I could update the last relation twofold, using the first two ones like: $$yxy^2=x^2\qquad\text{or}\qquad xyx=y$$ The last one puzzles me a bit from a "logical point of view", since I cannot "see" how $y$ could stay invariant when wrapped around two $x_s$.
Have you any hints about how to check that? I mean, an I forced to enumerate some possible group elements and search a contradiction among them, or could I found a more elegant solution finding a contradiction playing with relations only?
Thanks in advance
Why would we guess a group of order (at most) $9$ from the presentation $$\langle x, y : x^3 = y^3 = 1, yx = x^2y \rangle ?$$
Well, a bit of practice might help. Imagine you have a word in $x$ and $y$. By using the relation $x^3=1$, you can replace any negative powers of $x$ with positive powers (since $x^{-1}=x^2$), and reduce any exponent modulo $3$. Same for $y$. And by using the relation $yx=x^2y$, you can "shuffle" any $y$ that appears to the left of any $x$ to the right. After performing all suitable shuffles and reductions, you will end up with a word of the form $x^ay^b$, which is equal to whatever long word you originally had, and in which you have $0\leq a\leq 2$, $0\leq b\leq 2$. That is, one can tell from looking at the presentation (and with some experience) that whatever this group is, it has at most $9$ elements. It's possible it has exactly nine, but we have not established that.
No. That is not correct. We know $x^3=1$; we are not told explicitly whether $x=1$ or not. It's possible that you can conclude that, and this is not a "contradiction". If your book says explicitly that the relation $x^3=1$ implies the non-equality $x^2\neq 1$, then your book is using nonstandard notation and definitions; that is not how group presentations generally work. (If you have somehow infered this from the book, then please discard that inference, because it is wrong.)
In a group presentation, you get the relations you are given plus any relation that may be deduced from those relations. They do not imply any inequality. The presentations describes the group as a quotient of a free group: in this case, your group is the quotient of the free group on $x$ and $y$ by the normal subgroup generated by $x^3$, $y^3$, and $(yx)(x^2y)^{-1}$. You are describing the normal subgroup, and the fact that this normal subgroups contains these three elements does not tell you, by virtue of what those three elements are, that $x^2$ or $x$ do not lie in the normal closure of those elements. So the fact that you have the relation $x^3=1$ does not tell you, in and of itself, that you cannot have $x^2=1$ or $x=1$.
Now, groups of order $9$ are abelian. If this were a group of order $9$, then $yx=x^2y$ would also give $yx=yx^2$, and hence $x=x^2$, which implies $x=e$. That would mean that the group has order $3$, generated by $y$.
Without going into more details about how to try to establish an isomorphism type from a presentation (which is hard in general), let's consider that we already know that the group has order at most $9$, and show that your relations mean that the group must in fact be abelian. If it were nonabelian, then it is either of order $6$ or of order $8$, because those are the only orders less than or equal to $9$ which include nonabelian groups. But it cannot be of order $8$, because nontrivial elements in a group of order $8$ do not have exponent $3$ ($x^3=y^3=1$). And in a nonabelian group of order $6$, the only elements of order $3$ are powers of each other, so we would just be sitting inside the cyclic subgroup of order $3$ (and they do not satisfy $yx=x^2y$ in any iteration anyway). This tells us that the group cannot be nonabelian. So we are dealing with an abelian group.
Since it is necessarily abelian, we conclude that we can also deduce that $x=e$ as noted above, and so the presentation is that of the cyclic group of order $3$ generated by $y$.
If you wanted to play a bit more with the presentation instead, we might note after playing around a bit that $$y^2x=yyx = y(yx) = y(x^2y) = (yx)(xy) = (x^2y)(xy) = x^2(yx)y = x^2(x^2y)y = x^4y^2 = xy^2$$ which means that $x$ commutes with $y^2=y^{-1}$; therefore, $x$ commutes with $y$. Therefore, we have an abelian group, and the rest of the analysis to determine that $x=1$ follows.