The following classic exercise question: show that the group $G$ is metabelian iff $G'' = e$ ($G''$ denotes $(G')'$, the commutator subgroup of the commutator subgroup of $G$).
I know that the commutator group $G$ is denoted as: $[a, b]=a^{-1}b^{-1}ab$ for any $a, b \in G$. What I would like to know is how do I express "the commutator subgroup of the commutator subgroup of $G$" in mathematical notation. I know there are identities for commutator subgroups, but I am not able to find the relevant ones for answering my question. I would appreciate it if someone can give some clarification to this matter. Thank you in advance
The commutator subgroup $G'$ is generated by the commutators, elements of the form $[x,y]=x^{-1}y^{-1}xy$ with $x,y\in G$. That is, $$G' = \langle [x,y]\mid x,y\in G\rangle.$$ So the elements of $G'$ are products of commutators (since $[x,y]^{-1}=[y,x]$, inverses of commutators are themselves commutators). You have to go fairly far afield before you hit a group in which the set of commutators is not equal to the commutator subgroup: the smallest examples are two groups of order $96$ (this was determined by Robert Guralnick in his doctoral dissertation); for $p$-groups, I believe the smallest examples are of order $p^6$ for odd $p$, and $2^7$ for $p=2$.
The commutator subgroup of $G'$, $G''$, is then the subgroup generated by commutators $[x,y]$ with $x,y\in G'$; so each of $x$ and $y$ could be a product of commutators, and not merely a single commutator.
Fortunately, we have some commutator identities that allow us to simplify this generating set: we have $$\begin{align*} [xz,y] &= [x,y]^z[z,y]\\ [x,zy] &= [x,y][x,z]^y. \end{align*}$$ Therefore, if $x$ and $y$ are elements of $G'$, then $[x,y]$ can be decomposed as a product of conjugates of commutators of the form $[c,d]$, where $c$ and $d$ are both "pure" commutators, elements of the form $[g,h]$ with $g,h\in G$. And since $[c,d]^g = [c^g,d^g]$, it follows that any element of the form $[x,y]$ with $x,y\in G'$ can be rewritten as a product of elements of the form $\bigl[ [g,h],[r,s]\bigr]$, where $g,h,r,s\in G$. As each of these elements is itself in $G''$, it follows that $$G'' = \bigl\langle \bigl[ [x,y],[z,w]\bigr]\bigm| x,y,z,w\in G\bigr\rangle.$$ Similar simplifications can be done for the further terms of the derived series. Define $$\begin{align*} K_0 &= G;\\ K_1 &= \{ [x,y]\mid x,y\in K_0\};\\ K_2 &= \{ [x,y]\mid x,y\in K_1\};\\ &\vdots\\ K_{n+1} &= \{ [x,y]\mid x,y\in K_{n}\};\\ &\vdots \end{align*}$$ Then the derived series of $G$, defined by $$\begin{align*} G^{(1)} = G'=[G,G]&=\langle [x,y]\mid x,y\in G\rangle,\\ G^{(n+1)} = [G^{(n)},G^{(n)}] &= \langle [x,y]\mid x,y\in G^{(n)}\rangle, \end{align*}$$ can be shown to satisfy $G^{(n)} = \langle K_n\rangle$.
See also this math.overflow question and links therein, as well as this question in this site, both about groups in which the set of commutators is not equal to the commutator subgroup.