Prove: if $a,b\in G$ commute with probability $>5/8$, then $G$ is abelian

Question

Suppose that $G$ is a finite group. If $P( ab=ba ) >5/8$, prove $G$ is abelian.

Answer 1

Split into two cases: (i) $G/Z(G)\cong C_2\times C_2$ is Klein-$4$, and (ii) $Z(G)$ has index $\ge6$. Since the quotient $G/Z(G)$ cannot be nontrivial cyclic, these two cases cover all bases. Let $Z=Z(G)$.

$\rm\color{Blue}{(i)}$: Write $G=Z\sqcup aZ\sqcup bZ\sqcup cZ$. Check $C_G(g)=Z\cup aZ$ for $g\in aZ$, and similarly for the other two cosets $bZ$ and $cZ$, while of course $C_G(g)=G$ if $g\in Z$. Use $[\Omega]$ for the Iverson bracket (i.e. $[\Omega]=1$ if $\Omega$ is true, and $[\Omega]=0$ otherwise). Calculate

$$P=\frac{1}{|G\times G|}\sum_{a\in G}\sum_{b\in G}[ab=ba]=\frac{1}{|G|^2}\sum_{a\in G}|C_G(a)|=\frac{1}{|G|^2}\left(\sum_{a\in Z}|G|+\sum_{a\in G\setminus Z}\frac{|G|}{2}\right)$$

$$=\frac{1}{|G|^2}\left(|Z||G|+|G\setminus Z|\frac{|G|}{2}\right)=\frac{1}{|G|^2}\left(\frac{|G|}{4}|G|+\frac{3|G|}{4}\frac{|G|}{2}\right)=\frac{1}{4}+\frac{3}{4}\cdot\frac{1}{2}=\frac{5}{8}.$$

$\rm\color{Blue}{(ii)}$: Start the same way as above, writing

$$P=\frac{1}{|G|^2}\sum_{a\in G}|C_G(a)|=\frac{1}{|G|^2}\left(\sum_{a\in Z}|G|+\sum_{a\in G\setminus Z}|C_G(a)|\right).$$

Using $|Z|\le|G|/6$ and $|C_G(a)|\le|G|/2$, we obtain the upper bound

$$P\le\frac{1}{|G|^2}\left(|G||Z|+\big(|G|-|Z|\big)\frac{|G|}{2}\right)=\frac{1}{|G|^2}\left(\frac{|G|^2}{2}+|Z|\frac{|G|}{2}\right)\le\frac{1}{2}+\frac{1}{6}\cdot\frac{1}{2}=\frac{7}{12}.$$

Note $7/12<5/8$. Therefore, we have proven:

Theorem. $P>\frac{5}{8}$ iff $G$ is abelian, $P=\frac{5}{8}$ iff $G/Z(G)$ is Klein-$4$, and $P\le\frac{7}{12}$ otherwise.

There are a couple slick character-theoretic proofs on mathoverflow, as well as forcing probabilities for solvability, nilpotence and odd cardinality. Note the $5/8$ bound is true for compact topological groups, where we pick $(a,b)\in G\times G$ according to the normalized Haar measure. One formula for the commuting probability $P$ that I didn't discuss above is as follows:

$$P=\frac{1}{|G|^2}\sum_{a\in G}|C_G(a)|=\frac{1}{|G|}\sum_{a\in G}\frac{1}{[G:C_G(a)]}=\frac{1}{|G|}\sum_{i=1}^k\sum_{a\in C_i}\frac{1}{|C_i|}=\frac{1}{|G|}\sum_{i=1}^k\frac{|C_i|}{|C_i|}=\frac{k}{|G|},$$

where $C_1,\cdots,C_k$ are the conjugacy classes of $G$ and $k$ is the number of conjugacy classes.