Here is an incomplete proof:
If $H\le G$ is a subgroup, there are only three possibilities for its cardinal: $1,p$ or $p^2$ by Lagrange.
If $|H|=p^2$ then there exists $g\in H\subset G$ of order $p^2$ and so $G$ is cyclic.
If $|H|=1$ and there is no other proper subgroup then $G$ is cyclic.
So there must be a subgroup of order $p$, let $|H|=p$. Let $x$ be in $G$. We will show that $xHx^{-1}=H$ and so $H\triangleleft G.$ By contradiction let's suppose $xHx^{-1}\ne H$.
$xHx^{-1}$ is a subgroup of $G$ because $$(xhx^{-1})^{-1}=xhx^{-1}\in xHx^{-1} \text{ and }~ xhx^{-1}xh'x^{-1}=xhh'x^{-1}\in xHx^{-1}~\forall h,h'\in xHx^{-1}$$
There is a formula for the cardinality of a product of groups that says $$\begin{align}|xHx^{-1}|\cdot|H|&= |xHx^{-1}H|\cdot|xHx^{-1}\cap H|\\ |(xHx^{-1})H| &= \frac{|xHx^{-1}|\cdot|H|}{|xHx^{-1}\cap H|}=\frac{p\cdot p}{1}\end{align}$$
And here I'm not sure how to show that $xHx^{-1}$ has cardinality $p$, I guess for the same reason as $H$.
And why is the intersection trivial?
If the equality above is true we get that $xHx^{-1}H=G$ so for any $x\notin H$ there exist some $h_1,h_2\in H$ such that $x=xh_1x^{-1}h_2$
$$\begin{align}e &=h_1x^{-1}h_2\\ h_1^{-1}&= x^{-1}h_2\\ h_1^{-1}h_2^{-1}&= x^{-1}\implies x\in H \text{ (impossible)}\end{align}$$
Once we arrive at a contradiction we have that $xHx^{-1}=H$ so $H$ is normal in $G$.
Then I have to show that there exists $K\le G$ such that $K\cap H=\{e\}$ and $K\triangleleft G$ and from this it will be easy to show that $G\cong K\times H$
You can argue like this.
If there were an element of $G$ if order $p^2$, then $G$ were cyclic, generated by this element; thus, every non-identity element of $G$ has order $p$. Fix arbitrarily a non-identity element $g_1\in G$, and then an element $g_2\in G$ not lying in the subgroup generated by $g_1$. For $i\in\{1,2\}$, denote by $G_i$ the subgroup generated by $g_i$. Both these subgroups are cyclic; hence, generated by any of their non-identity elements. It follows that the intersection of $G_1$ and $G_2$ is trivial (any non-identity element of the intersection would generate both $G_1$ and $G_2$, but $G_1\ne G_2$). It follows that all the $p^2$ products $h_1h_2$ with $h_1\in G_1$ and $h_2\in G_2$ are pairwise distinct. Thus, $G$ is exactly the set of all these products, as wanted.