I am reading a paper by William Carter titled "New examples of torsion-free non-unique product groups" and saw the following group:
$$P_k=\langle a,b\mid ab^{2^k}a^{-1}b^{2^k},ba^{2}b^{-1}a^{2} \rangle.$$
I am quite new to group theory, so I have a tough time understanding this notation. I understand that the group presentation is usually given by $\langle S|R\rangle$ where $S$ are the generators of the group and $R$ consists of the relations among the given generators. However, the relations here are the part that I do not understand. Usually when seeing for example a dihedral group the relations are given by $x^2=1$, $y^n=1$ and so on. What do the relations here mean?
They are called defining relators and are used in the definition of the group defined by a presentation: the presentation $\langle S \mid R \rangle$ defines the group $F(S)/\langle S^R \rangle$, where $F(S)$ is the free group on the set $S$.
A relation $u=v$ is equivalent to a relator $uv^{-1}$, so the presentation in your example could be written equivalently as $$\langle a,b \mid ab^{2^k}a^{-1}b^{2^k}=ba^{2}b^{-1}a^{2}=1 \rangle,$$ or alternatively $$\langle a,b \mid ab^{2^k}a^{-1}= b^{-2^k},\,ba^{2}b^{-1}=a^{-2} \rangle,$$