Let $p$ be an odd prime such that $q:=2p+1$ is also prime.
Denote $g(p):=gnu(2pq^2)$ = number of groups of order $2pq^2$ upto isomorphy.
The following table shows the first few values
p q g(p)
3 7 23
5 11 27
11 23 39
23 47 63
29 59 75
41 83 99
The calculation with GAP quickly becomes time-consuming, so my question :
Is a formula for $g(p)$ known ?
The Sylow-theorem tells us that a group of order $2pq^2$ must have a normal subgroup of order $q^2$. Does this help to get a formula ?
Following your thoughts. Let G be a group of order $2pq^2$ and $S$ be its unique $q$-Sylow. Then $Q:=G/S$ is a quotient group of order $2p$.
With some basic group theory, there are only two choices for $S$ :
$$S_1:=\mathbb{Z}/q^2\text{ and } S_2:=(\mathbb{Z}/q)^2 $$
and only two choices for $Q$ :
$$Q_1:=\mathbb{Z}/2p\text{ and } Q_2:=D_{2p}\text{ (the dihedral group)}$$
Furthermore, we can use the Schur-Zassenhaus theorem to state that $G$ is isomorphic to $S\rtimes_{\phi}Q$ where $\phi:Q\rightarrow Aut(S)$ is a group morphism.
Now, you just have to compute, for each choice of $(i,j)$, the number of different groups (up to isomorphism) given by $S_i\rtimes_{\phi}Q_j$ where $\phi:Q_j\rightarrow Aut(S_i)$ is a group morphism.
For instance for $(i,j)=(1,1)$, I have four different isomorphism classes and $(i,j)=(1,2)$, I have two different isomorphism classes. Indeed, when $i=1$, we have that $Aut(S_1)$ is cyclic of order $2p(2p+1)$.
When $j=1$, $Q_1$ is generated by one element (let us say it is $\gamma$). If $\phi(\gamma)$ is trivial, we have one isomorphism class. If $\phi(\gamma)$ is of order $2$ then there is a unique element of order $2$ in $Aut(S_1)$, whence a unique isomorphism class. If $\phi(\gamma)$ is of order $p$ then we get $p-1$ choices for $\phi(\gamma)$ but they all lead to the same isomorphism class for the semi-direct product. If $\phi(\gamma)$ is of order $2p$ then we get $p-1$ choices for $\phi(\gamma)$ but they all lead to the same isomorphism class for the semi-direct product. Whence we have four choices.
If $j=2$ then we know that the abelianization of $Q_2$ is $\mathbb{Z}/2$, it follows that there are exactly two group morphisms from $Q_2$ to $Aut(S_1)$ (one for which the image of $\phi$ is trivial and one for which the image of $\phi$ is of order $2$).
Maybe you can handle the more difficult case where $i=2$ (the automorphism group of $S_2$ is more complicated...).