Integrating directly: $$\int xydx = y \int x dx = y\left(\frac{x^2}{2}\right) = \frac{1}{2}x^2 y$$
Now let's instead integrate using the integration by parts formula: $$\int u dv = uv - \int v du$$
We have: $$\int xydx = x^2y-\int x d(xy) = x^2y- \int x(xdy+ydx) = x^2 y-\int x^2 dy-\int xy dx = x^2y-x^2y-\frac{1}{2}x^2y = -\frac{1}{2}x^2 y$$
Which is almost the right answer, except that the sign is wrong.
Question. What's going on here?
$y$ is constant here. So $dy=0$. At least that is the way you treat it in the first integration.