I want to show that the defining relations $a^m=b^n=e, ab=ba^k$ define a group of order $mn$ with a normal subgroup of order $m$, if $k^n \equiv 1 \pmod m$.
Consider the set of symbols of the form $b^q a^r$, where $q \in \{0,1,\ldots,n-1\}, r \in \{0,1,\ldots,m-1\}$. This set forms a group: Closure holds because $(b^q a^r)(b^s a^t) = b^{q+s} a^{rk^s +t }$. It follows that the inverse of $b^q a^r$ is $b^s a^t$ where $s=-q=n-q$ and $t=-rk^s=m-rk^{n-q}$. Associativity can also be verified. Finally, let $A:=\{a^r: r=0,1,\ldots,m-1\}$. Then $A$ is a normal subgroup since $(b^q a^r)^{-1} a^s (b^q a^r) = a^{-r} b^{-q} a^s b^q a^r = a^{-r} b^{-q+q} a^{sk^q +r} = a^{sk^q} \in A$.
My question is on why the assumption $k^n \equiv 1 \pmod m$ is necessary? (It is mentioned in this exercise, but I don't see where I used it in the proof.)
Prove by induction on $t$ that $b^{-1}ab = a^k \Rightarrow b^{-t}ab^t = a^{k^t}$ for all $t \ge 0$, then putting $t=n$ gives $a = a^{k^n}$, so $k^n \equiv 1 \bmod m$.