This is Exercise 2.1.4(b) of "Combinatorial Group Theory: Presentations of Groups in Terms of Generators and Relations," by W. Magnus et al.
The Question:
Let $F=\langle a, b\rangle$. If $N$ is the normal subgroup of $F$ generated by each of the following sets of words, find the index of $N$ in $F$:
(b) $ab, ab^{-1}$.
My Attempt:
Following this answer, we have that $F/ N$ is generated by $a'=aN, b'=bN$, and $a'b'^{-1}=1$ gives $a'=b'$, while $a'b'=1$ then gives $a'^2=1$, so that $F/N=C_2$.
Under the homomorphism $f: F\to \Bbb Z/2\Bbb Z$ given by $f(a)=f(b)=1+2\Bbb Z$, we have $ab, ab^{-1}\in \ker f$ and so $N\subseteq \ker f$. But $\ker f$ has order $2$ in $F$. Hence the index of $N$ in $F$ is $2$.
Is this right? I think I'm missing a few steps.
Please help :)
I am not sure why you keep going after $F/N=C_2$. The index of $N$ in $F$ is the number of cosets of $N$. That is, $\lvert F/N\rvert=\lvert C_2\rvert =2$.
Another thing you can do is manipulate a group presentation. $F/N=\langle a,b\mid ab,ab^{-1}\rangle$. Since $ab^{-1}=1$ in this quotient, $a=b$, so we may eliminate $b$ to get the presentation $\langle a\mid aa,aa^{-1}\rangle=\langle a\mid a^2\rangle=C_2$.
Yet another thing you can do is look at the left-multiplication-action of $F$ on the set of cosets $F/N$. Since $N$ is normal, $gabg^{-1}$ and $gab^{-1}g^{-1}$ are in $N$ for all $g\in F$, and so $gabg^{-1}N=N$ and $gab^{-1}g^{-1}N=N$ (and corresponding statements for the inverses of $ab$ and $ab^{-1}$). We see $bN=b(b^{-1}ab^{-1}b)N=aN$ and $a^2N=a(aN)=a(bN)=abN=N$.
Claim: $N,aN$ are the only cosets, and they are distinct. We have already seen that $a$ permutes them and that $bN=aN$. The last check for finding all the cosets is $b(aN)=ba(b(ab)^{-1}b^{-1}N)=babb^{-1}a^{-1}b^{-1}N=N$. Distinctness follows from $ab(aN)=N$ and $ab^{-1}(aN)=N$, basically through your $\mathbb{Z}/2\mathbb{Z}$ argument with $N$ and $aN$ standing in for $2\mathbb{Z}$ and $1+2\mathbb{Z}$.
These last two paragraphs make more sense to me through Schreier graphs to do the Todd-Coxeter algorithm. https://www.math.cornell.edu/~kbrown/7350/toddcox.pdf (Though, since $N$ is normal, it is actually about constructing the Cayley graph of $F/N$ and counting the vertices.)