Let $G$ be generated by ${a,b}$ with relations $a^8 = b^2a^4 = ab^{-1}ab = e$. I want to show that $G$ has order at most 16.
What's a good way of solving problems of this kind? Let $N$ be the normal subgroup of $F_{a,b}$ (free group on $\{a,b\}$) generated by $\{a^8,b^2a^4,a^{-1}ab\}$. My idea is to show the existence of an injection between $F_{a,b}/N\to H$ where $H$ is some group with order 16.
One approach is brute force: you've got elements $e, a, \cdots, a^7$ and $b$ times each of those (i.e., $b, ba, \cdots, ba^7$). I'll show that every word in $F(a, b)$ is equivalent to one of those, so that the subgroup generated by your relations has at most 16 cosets:
Your last rule says that $ab = ba^{-1}$ or that $a^{-1}b = ba$. So you can use that to always move all the $b$s to the front. And the first rule says that $a$ never has an exponent greater than 8. And the middle rule says that you can replace any $b^2$ by an $a^{-4}$, so you never need to have more than one $b$ at the front.
To make this concrete, here's an example of reducing a long word. Rule 1 denotes the $a^8$ relation; rule 2 the $b^2 a^4$ relation, and Rule 3 the $ab^{-1}ab$ relation.
\begin{align} abbabaaabab &= (ab)babaaabab \\ &= b(a^{-1}b)abaaabab & \text{rule 3}\\ &= bba(ab)aaab(ab) &\text{rule 3}\\ &= bba(ba^{-1})aaab(ba^{-1}) & \text{rule 3 twice}\\ &= (b^2)(ab)a^{-1}aaa(b^2)a^{-1} &\text{regrouping}\\ &= (a^{-4})(ba^{-1})a^{-1}aaa(a^{-4})a^{-1} &\text{rule 2 twice, rule 3 once}\\ &= a^{-4}ba^{-2}aaaa^{-5} &\text{simplifying exponents}\\ &= a^{-4}ba^{-4} &\text{simplifying exponents}\\ &= a^{-3}(ab)a^{-4}& \text{regrouping}\\ &= a^{-3}(ba^{-1})a^{-4} &\text{rule 3}\\ &= a^{-3}ba^{-5} &\text{regrouping}\\ &...\\ &= b a^{-8} &\text{repeating last 3 ops several times}\\ &= b &\text{rule 1}. \end{align} Conclusion: there are at most 16 cosets of the group generated by your four relations.