Theorem. Let $G$ be a finite, non-abelian $p$-group all of whose proper subgroups are abelian. Then $|G'|=p$.
Take a counterexample of minimal order. Assume that exist a $H$ such that $1<H<G'$.
Then (by $G'\leq \Phi (G) \leq Z(G)$) $H\vartriangleleft G$. From this we deduce we can assume $|G'|\leq p^2$.
Then? How am I supposed to continue?
Edit
Additional infos
$G'$ is elementary abelian since $G$ is Frattini-in-center.
I was thinking... $G$ is a finite, nilpotent (so also soluble) group; so there exist a $G_1\vartriangleleft G$ such that $|G:G_1|=p$. $G$ is minimal non abelian, hence $G=<x, y>$. We can suppose $y\notin G_1$, but then there exist a $g\in G_1$ such that $y^n=xg$; so $G=<y,g>$. Then, as above, $G'=[G_1,y]$ and every $x\in G'$ is a product of element of the form $[y^{n_1}g^{m_1}...y^{n_t}g^{m_t}, y^c]$. Hence every $x\in G'$ has the form: $[g^m, y^s]=[g, y]^{k}$.
$G'$ is cyclic and elementary abelian, since $G$ is Frattini-in-center, so $|G'|=p$.