Is there a finite $p$-group $G$ such that $G$ has less number of automorphisms than some subgroup: $$|\mbox{Aut}(G)|<|\mbox{Aut}(H)| \mbox{ for some }H\leq G.$$ If there is such a group, then can this happen in some abelian $p$-group?
Perhaps there are non-nilpoent groups $G$ satisfying above condition, since, long back, I had seen an exercise to provide such example in Problems in Group Theory by John D. Dixon
The example of a non-nilpotent group $G$ and its $2$-subgroup is here. But, my question is within the Category of $p$-groups.
If $H = C_2^3$ (elementary abelian of order $8$) then ${\rm Aut}(G) ={\rm GL}(3,2)$, which has order $168$.
But $G = D_8 \times C_2$ ($D_8$ means dihedral of order $8$) contains $H$ as a subgroup, and $|{\rm Aut}(G)| = 64$. I did that by compoter calculation, but it's not hard to prove by hand. The generator of $D_8$ of order $4$ can map uner an automorphism to any element of order $4$ and there are $4$ such. A generator of $D_8$ of order $2$ can map to any non-central element of order $2$ and there are $8$ such. Finally, a generator of $C_2$ maps to a central generator of order $2$ that is not contained in $D_8$, and there are $2$ such. So $|{\rm Aut}(G)| = 4 \times 8 \times 2 =64$
There are similar examples at least for $p=3,5,7$.
I would guess that there are no abelian examples, but I might be wrong!