I was able to show that $\sum_{a^2 b^3 \leq x} 1 =\zeta(3/2)x^{1/2} + O(x^{1/3})$ by rewriting it as
$$ \sum_{b^3 \leq x} \sum_{a^2 \leq x/b^3} 1 = \sum_{b^3 \leq x}\left((\frac{x}{b^3})^{1/2}+O(1)\right)$$
but it looks like this simple approach won't work for the more complicated case. I was given a hint to use Dirichlet's hyperbola method, but I'm a bit unsure about how to use it. I know I would need to let
$f(n)=$ \begin{cases} 1 & \text{if } n=a^2 b^3 \\ 0 & \text{otherwise} \end{cases}
and then write $f$ as a convolution of two other multiplicative functions so that I could use DHM on $\sum_{n \leq x} f(n)$, but I cannot find these two functions. Is this the right approach?
So the first trick is to note that if $a^2 b^3 \leq x$, either $a \leq x^{1/5}$ or $b \leq x^{1/5}$. This gives
$$\sum_{a^2 b^3 \leq x}1 = \sum_{a \leq x^{1/5}} \sum_{b \leq x^{1/3}/a^{2/3}}1 + \sum_{b \leq x^{1/5}} \sum_{a \leq x^{1/2}/b^{3/2}}1 - \sum_{a \leq x^{1/5}} \sum_{b \leq x^{1/5}}1$$
$$ =\sum_{a \leq x^{1/5}} \left( (x/a^2)^{1/3} +O(1) \right) + \sum_{b \leq x^{1/5}} \left( (x/b^3)^{1/2} +O(1) \right) -(x^{1/5}+O(1))^2. $$
Factoring out the powers of $x$ and summing the $O(1)$ to become $O(x^{1/5})$, the next trick is to use the following formula on the two resulting sums:
$$\sum_{n \leq x}\frac{1}{n^s} = \frac{x^{1-s}}{1-s} + \zeta(s) + O(x^{-s})$$
and then it is just a matter of directly computing to give the desired asymptotic.