By using the Sylow Theorems we need $n_5\in \{1,2^2\cdot 19\}$ and $n_{19}\in\{1,20\}$.
First Case: If $n_5=2^2\cdot 19$ and $n_{19}=20$, the group cannot stand the pressure: we have too many elements. Indeed, because $5$ and $19$ are prime, the Sylow groups intercept trivially and there are $2^2\cdot 19\cdot 4$ elements of order $5$ and $20\cdot 18$ elements of order $19$. But this yields a contradiction: $|G|=2^2\cdot 5 \cdot 19<2^2\cdot 19\cdot 4+20\cdot 18$.
Second Case: Say $n_5=1$ and $n_{19}=20$. Take $P\in \text{Syl}_5 G$ and $Q\in \text{Syl}_{19}G$. We have $Q\cap P=1$ and $P\unlhd G$ therefore $P\rtimes Q\leq G$. I wanted to prove normality (haven't been able to prove it thus far) $P\rtimes Q \unlhd G$. Because if it were normal, we would have:
$$gQg^{-1}\leq gP\rtimes Q g^{-1}=P\rtimes Q$$
Thus all Sylow-19 groups would be there and we would need $20\cdot 18$ elements of order $19$ there. But the size of $P\rtimes Q$ is only $ 19\cdot 5$. Absurd.
Third Case: I think we reach a similar contradiction by assuming $n_5=2^2\cdot 19$ and $n_{19}=1$. Indeed, we would have $Q\rtimes P\leq G$. And if we assume normality, all Sylow $5$ subgroups would be in the semidirect product. Thus we would have $2^2\cdot 19\cdot 4$ elements of order $5$ in a group of order $19\cdot 5$.
Basically I need to prove the semidirect product is normal in the second and third cases.
Jyrki Lahtonen and Sean Eberhard have solved this in two lines in the comment section, but encouraged me to write a more detailed answer.
We already know $n_5=1$ or $n_{19}=1$. Suppouse by ways of contradiction that $n_{19}=1$ and $n_5=2^2\cdot 19$, we have that $\exists ! P\in \text{Syl}_{19}G$, so $P\lhd G$. Let us take $Q\in \text{Syl}_{5}G$ and consider the semidirect product $P\rtimes Q$.
The semidirect product has four lateral classes $[G: P\rtimes Q]=|G|/|P\rtimes Q|=4$. Take $U \in \text{Syl}_2 G$ we know $U=\{g_0,g_1,g_2,g_3\}$. If $g_iP\rtimes Q=g_jP\rtimes Q$ we have $g_j^{-1}g_i\in P\rtimes Q \Rightarrow g_j^{-1}g_i=1$ because $g_j^{-1}g_j \in (P\rtimes Q)\cap U$ and therefore it has order dividing $4$ and also $19\cdot 5$. Therefore $g_j P\rtimes Q$ are all lateral classes and $U (P\rtimes Q)=G$ where we consider the usual set multiplication. Indeed :$$G=\dot{\cup}_{i=0}^3 g_i Q\rtimes P=U(P\rtimes Q)$$
Let $g\in Q$, clearly $\phi_g|_P(x)=gxg^{-1}$ is an automorphism $\phi_g|_P:P\rightarrow P$ (because $P\lhd G$ )such that $|\phi_g|$ divides 5 (because $\phi_g^5(x)=g^5xg^{-5}=x=I(x)$).
We know $P\approx \mathbb{Z}_{19}$. This means an automorphism $\psi$ is well defined if we know to what generator $1$ is taken. In other words we are interested in knowing $\psi_j(1)=j$ such that $0<j<19$. Also:
$$\psi_j\psi_k(1)=\psi_j (k)=k\psi_j(1)=kj=\psi_{jk }$$
So automorphisms are actually $\mathbb{Z}_{19}^*$ in disguise. But that is a group of order $18$. So $\phi_g|_P$ has an order dividing $18$ and $5$ and hence it must be trivial. In other words, $gxg^{-1}=x$ for $\forall x\in P$ and $\forall g\in Q$. We will actually need this in the form $g=xgx^{-1}$ $\forall x\in P$ and $\forall g\in Q$.
Let us compute conjugacy classes for $Q$ by taking an arbitray $g\in G=U(P\rtimes Q)$:
$$gQg^{-1}=upqQq^{-1}p^{-1}u^{-1}=uQu^{-1}$$
All conjugacy classes are determined by this $u$ term. But there are only four possibilities for it $(|U|=4)$. This means $n_5\leq 4$. Contradiction.