I'm having trouble with this exercise:
Let $G$ be the free group generated by $a$ and $b$. Prove that the commutator subgroup $G'$ is not finitely generated.
I found a suggestion that says to prove $G'$ is generated by the collection $\{[x^m,y^n]\mid m,n\in\mathbb{Z}\}$. I don't know how to prove this, and how it helps.
If you could please provide me with some hints... Thanks!
On
You can prove this using algebraic topology$^{\dagger}$: in fact, it follows from an exercise in Hatcher's book (see this related math.SE question). Basically: use Reidemeister-Schreier. I have written out a "full-ish" proof - but if you are comfortable with covering spaces then this proof is ridiculously quick and elegant!
The covering-spaces proof that $F_2^{\prime}=[F(a, b), F(a, b)]\cong F_{\infty}$ is as follows: The free group $F(a, b)$ is the fundamental group of a bouquet of two circles:
Using $\widetilde{X}$ to denote the universal cover, by standard covering-space theory the derived subgroup $F_2^{\prime}=[F(a, b), F(a, b)]$ is isomorphic to the fundamental group of $\widetilde{X}/F_2^{\prime}$. Now, $\widetilde{X}$ is just the Cayley graph of $\langle a, b; -\rangle$,
while $\widetilde{X}/F_2^{\prime}$ is the Cayley graph of $\mathbb{Z}\times\mathbb{Z}$,
Then, $\pi_1(\widetilde{X}/F_2^{\prime})$ is the group with presentation $\langle V; R\cup T\rangle$ where $V$ is the set of $1$-cells (edges) of $\widetilde{X}/F_2^{\prime}$, $R$ is the set of $2$-cells of $\widetilde{X}/F_2^{\prime}$, and $T$ is a spanning tree for $\widetilde{X}/F_2^{\prime}$. Now, $\widetilde{X}/F_2^{\prime}$ has no $2$-cells, and taking any spanning tree we are still left with infinitely many edges (for example, take the tree $T=\bigcup_{y\in\mathbb{Z}}\{(x, y): x\in\mathbb{Z}\}\cup\{(0, x): x\in\mathbb{Z}\}$). Hence, we have an infinitely generated group with no relations, so $F_2^{\prime}=[F(a, b), F(a, b)]\cong F_{\infty}$ as required.
In fact, there is nothing special about $\mathbb{Z}\times\mathbb{Z}$ here:
Thereom. Let $G$ be an infinite group. If $\phi: F_n\rightarrow G$ is a surjection from the free group of rank $n$, $\infty>n>1$, to $G$ then $\ker(\phi)$ is either trivial or is not finitely generated.
Proof. Suppose $\ker(\phi)$ is non-trivial, and write $N:=\ker(\phi)$. Then, similar to the above, $\widetilde{X}/N$ is a graph; it is essentially the Cayley graph for $G$ given by the generating set $\operatorname{im}(\phi(X))\subset G$ (note the subtlety if there are $x_1, x_2\in X$ such that $\phi(x_1)=\phi(x_2)$). Now, as above, $N=\pi_1(\widetilde{X}/N)$ is the group with presentation $\langle V; R\cup T\rangle$ where $V$ is the set of $1$-cells (edges) of $\widetilde{X}/N$, $R$ is the set of $2$-cells of $\widetilde{X}/N$, and $T$ is a spanning tree for $\widetilde{X}/N$. As $\widetilde{X}/N$ is a graph it has no $2$-cells. It remains to show that for any spanning tree $T$ of $\widetilde{X}/N$ there are infinitely many edges $e$ in $(\widetilde{X}/N)\setminus T$. As $\ker(\phi)$ is non-trivial there exists a non-empty word $W\in F(X)$ such that $\phi(W)=1$. This corresponds to a loop in $\widetilde{X}/N$, and indeed at each vertex $v$ there exists such a loop $\mathcal{l}_{W, v}$ of length $d:=|W|$. Note that the tree $T$ cannot contain every edge of the loop $\mathcal{l}_{W, v}$. As $G$ is infinite there exists an infinite sequence of pairwise non-equal vertices $v_1, v_2, \ldots$ of $\widetilde{X}/N$ such that no two vertices $v_i, v_j$ are linked by a path of length $\leq2d$. Therefore, the loops $\mathcal{l}_{W, v_i}$ and $\mathcal{l}_{W, v_j}$ contain no edges in common. Hence, there are infinitely many edges $e$ in $(\widetilde{X}/N)\setminus T$. We therefore have an infinitely generated group with no relations, so $N$ is not finitely generated as required.
$^{\dagger}$ Serios recently reminded me of this proof in the comments to this fine math.SE answer.
On
Another argument based on algebraic topology applied to graphs:
Property: Let $F$ be a free group of finite rank $k$ and $H \lhd F$ be a nontrivial normal subgroup. If $H$ is finitely generated then it is a finite-index subgroup.
Now, it is possible to deduce that $D(\mathbb{F}_2)$ is not finitely-generated for instance by noticing that the quotient $\mathbb{F}_2/ D(\mathbb{F}_2)$ (namely the abelianization) is isomorphic to $\mathbb{Z}^2$ (and in particular is infinite).
Proof. $F$ may be viewed as the fundamental group of a bouquet $B$ of $k$ circles. Let $\hat{B} \to B$ be the covering associated to the subgroup $H$. Because $H \lhd F$, the covering is normal, that is the restriction of the action $F= \pi_1(B) \to \mathrm{Aut}(\hat{B})$ on the fiber over the base point is transitive. Since $B$ has only one vertex, we deduce that $\mathrm{Aut}(\hat{B})$ acts transitively on $\hat{B}$.
From now on, suppose by contradiction that $H$ is an infinite-index subgroup, that is $\hat{B}$ is infinite.
Because $H \neq \{1\}$, $\hat{B}$ contains a cycle $C$. Now let $v_0,v_1,\ldots \in \hat{B}$ be a sequence of vertices such that $v_0 \in C$ and $d(v_i,v_j) > 2 \mathrm{lg}(C)$, and let $\varphi_1, \varphi_2, \dots$ be a sequence of automorphisms of $\hat{B}$ such that $\varphi_{i+1}(v_i)=v_{i+1}$. Then $\varphi_1(C), \varphi_2(C),\dots$ is a sequence of disjoint cycles in $\hat{B}$ so that $H= \pi_1(\hat{B})$ cannot be finitely-generated. $\square$
Nota Bene: From Nielsen-Schreier formula, we know that there is an equivalence: a nontrivial normal subgroup is finitely-generated if and only if it is a finite-index subgroup.
More information may be found in the note Graphs and Free Groups and in references therein. The question When is $G \ast H$ solvable? is also relevant.
On
The answers given thus far are great, but no one has posted my favorite proof of this result yet. Consider the homomorphism $\phi\colon F_2\to \mathbb{Z}^2$ defined on the generators by $a\mapsto (1,0)$ and $b\mapsto (0,1)$. This is just the quotient map $F_2\to F_2/[F_2,F_2]$, but we're going to reinterpret it in a slightly different way. For any $w\in F_2$, $$ \phi(w)=\left(\#\{a\text{'s in }w\}-\#\{\bar{a}\text{'s in }w\},\#\{b\text{'s in }w\}-\#\{\bar{b}\text{'s in }w\}\right)$$
Don't think of $\phi$ as the quotient by the commutator subgroup, just think of it as a funny map (you should check that it's a homomorphism) defined on the Cayley graph of $F_2$ as described above. Let's think about the $G=\ker \phi$ and suppose $G$ were finitely generated $G=\langle w_1,\ldots,w_k\rangle$. Think about the values $\phi$ takes on as you write out each $w_i$ in terms of $a$'s and $b$'s (or equivalently as you walk from the identity to $w_i$ in the Cayley graph). $\phi$ will increase, then decrease, then increase, then decrease, and so on until you reach $w_i$ where $\phi(w_i)=(0,0)$. The point is that in each coordinate of $\mathbb{Z}^2$, there will be some maximum value above which you never passed. This is true for each generator individually, so there is some big compact set in $\mathbb{Z}^2$ which you never leave when writing an individual generator.
Now write a word in your finite generating set of $G$, $w=w_{i_1}w_{i-2}\cdots w_{i_j}$, then write out that word in $a$'s and $b$'s and think about the values $\phi$ takes along the way. You will be stuck inside the same compact set in $\mathbb{Z}^2$ because each generator only makes it so far then returns to the origin.
To finish, you just say $\phi([a^n,b^n])\in G$, but to write this out in the Cayley graph of $F_2$, you must pass through vertices where $\phi$ is arbitrarily large. Thus, the finite set of $w_i$'s did not actually generate $G$.
Some ideas for you to work on:
An element of $\;F:=F(a,b)\;$ in normal form
$$\;w=a^{n_1}b^{m_1}\cdot\ldots\cdot a^{n_k}b^{m_k}\;,\;\;n_i,m_i\in\Bbb Z\;,\;\;n_1,m_k\in\Bbb Z$$
belongs to $\;F'\;$ iff the exponent sum of both letters is zero, meaning:
$$w\in F'\iff\sum n_i=\sum m_i=0$$
For example, $\;a^{-2}bab^{-1}a\in F'\;$ , but $\;a^{-1}ba^2b^{-1}\notin F'\;$ .
From the above, it is clear that $\;F'=\left\langle a^{-n}b^{-m}a^nb^m\;:\;\;{n\,,\,m\in\Bbb N}\right\rangle\;$ . To prove these are free generators you can try either to show that any normal word in those generators is the trivial element iff it is the void element (this seems to be a fairly non-so-hard approach), or perhaps the universal property of free groups.