When is the automorphism group of a nilpotent group nilpotent?

Question

Let G be a group. I know that if $Aut(G)$ be nilpotent then $G$ is nilpotent also. Since

$\frac{G}{Z(G)}‎\cong‎ Inn(G)‎\unlhd‎ Aut(G)$

Since $Aut(G)$ is nilpotent then $\frac{G}{Z(G)}$ is nilpotent. We have

$‎\gamma‎_n(\frac{G}{Z(G)})=Z(G)$

We can write

$‎\gamma‎_n(\frac{G}{Z(G)})=\frac{\gamma‎_n (G)Z(G)}{Z(G)}=Z(G)$

So, $\gamma‎_n (G)Z(G)‎\subseteq‎ Z(G)$ and $\gamma‎_n (G)‎\subseteq Z(G)$. Then G is nilpotent and $\gamma‎_{n+1} (G)‎=e$

I would like to know when the converse of my result is true? Clearly converse of my result not true always.

Please give me strong condition to converse of my result always be true.

Answer 1

There's actually a whole class of counter-examples that are quite easy to describe:

Let $G=C_p^n$ be the direct product of $n$ copies of the cyclic group of order $p$ with $p$ prime and $n\ge 3$ and denote the generator of the $i^{th}$ copy of $C_p$ by $x_i$. Then $S_n$ acts on $G$ through automorphisms defined by $\sigma(x_i)=x_{\sigma(i)}$ for $\sigma\in S_n$. This means $S_n$ is a subgroup of $\mathrm{Aut}(G)$ so, as $S_n$ is not nilpotent for $n\ge 3$, $\mathrm{Aut}(G)$ is not nilpotent.

Answer 2

It is known that the automorphism group of almost all finite $p$-groups is a $p$-group and hence nilpotent. See here. So for almost all finite nilpotent groups $G$, ${\rm Aut}(G)$ is nilpotent.

It is also conjectured that almost all finite groups are $2$-groups, so if that is true then ${\rm Aut}(G)$ is nilpotent for almost all finite groups.