I tried solving it by comparing it with $Mdx+Ndy =0$ for $M=y^2 +2x^2y$ and $N=2x^3 -xy$,
$$\frac{\partial M}{\partial y}=2y+2x^2,~~~~\frac{\partial N}{\partial x}=6x^2-y$$
$\displaystyle\frac{\partial M}{\partial y}\neq \frac{\partial N}{\partial x}$. I don't know how to procede further. Any help is appreciated
Let the integrating factor be $\mu=x^ay^b$,
$$x^ay^b(y^2+2x^2y)dx+x^ay^b(2x^3-xy)dy=0$$
we get
$$(b+2)x^ay^{b+1}+2(b+1)x^{a+2}y^b=2(a+3)x^{a+2}y^b-(a+1)x^ay^{b+1}$$
compare coefficients,
$$b+1=a+3,~~~b+2=-(a+1)$$
hence,
$$a=-\frac32, ~~b=\frac12$$
After multiplying the integral factor $$\mu=x^{-\frac32}y^{\frac12}$$ it becomes exact and you can proceed to solve it.