I am trying to solve $$ (x^2y^3+y)dx+(x^3y^2-x)dy=0 $$ First of all we check this equation for exactness
$\frac{\partial P}{\partial y}=3y^2x^2+1$
$\frac{\partial Q}{\partial x}=3x^2y^2-1$
The partial derivatives are not equal to each other. Therefore, this equation is not exact.
Then I calculate the difference of the derivatives: $\frac{\partial P}{\partial y}-\frac{\partial Q}{\partial x}=2$
Now I try to use the integrating factor in the form $z=xy$
$\frac{\partial z}{\partial x}=y, \frac{\partial z}{\partial y}=x$
Then:
$Q\frac{\partial z}{\partial x}-P\frac{\partial z}{\partial y}=y(x^3y^2-x)-x(x^2y^3+y)=-2xy$
and hence I get
$\frac{1}{\mu}\frac{\partial \mu}{\partial z}=\frac{\frac{\partial P}{\partial y}-\frac{\partial Q}{\partial x}}{Q\frac{\partial z}{\partial x}-P\frac{\partial z}{\partial y}}=\frac{2}{-2xy}=-\frac{1}{xy}$
Now I can find the integrating factor by integrating the last equation and I get that $\mu=-xy$
Multiplying both sides of the first equation with $-xy$ I get
$$ -xy(x^2y^3+y)dx+(-xy)(x^3y^2-x)dy=0 $$
And when I calculate $\frac{\partial P}{\partial y}$ and $\frac{\partial Q}{\partial x}$ they are still not equal to each other, so the equation is still not exact and I can't solve it. What am I doing wrong?
Let us solve the ODE: $$(x^2y^3+y)dx+(x^3y^2-x)dy=0~~~~(1)$$ The oP did a mistake the integrating factor should be $\mu=\frac{1}{xy}$, then multiplying (1) by $\mu$ we get $$\left (xy^2+\frac{1}{x} \right) dx+ \left(x^2y-\frac{1}{y} \right)dy=0$$ Now this exact ODE. Which can be rearranged as $$xy(ydx+xdy)+\frac{dx}{x}-\frac{dy}{y}=0 \implies\int xy d(xy)+\int \frac{dx}{x}-\int\frac{dy}{y}=C $$ $$\implies (xy)^2/2+\ln x-\ln y=C \implies x^2y^2+2\ln (x/y)=D$$