Use Green's Theorem to calculate the integral $\int_CP\,dx+Q\,dy$.
$P = xy, Q = x^2, C$ is the first quadrant loop of the graph $r = \sin2\theta.$
So, Green's Theorem takes the form $$\int\int_D \frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y} dA$$
$\frac{\partial Q}{\partial x} = 2x, \frac{\partial P}{\partial y} = x$
Based on the nature of the graph, polar coordinates are the way to go. I get limits of integration $\int_{\theta = 0}^{\pi/2} \int_{r=0}^{\sin2\theta}$. Using the fact that in polar coordinates $dA = rdrd\theta$, I arrive at the integral: $$\int_{\theta = 0}^{\pi/2} \int_{r=0}^{\sin2\theta} (2x-x)r\,dr\,d\theta$$ which simplifies to $$\int_{\theta = 0}^{\pi/2} \int_{r=0}^{\sin2\theta} r^2\cos\theta \,dr\,d\theta$$
The problem is, this integral doesn't seem to be solvable. I've even tried plugging it into an integral calculator, and gotten "cannot calculate this integral." What am I doing incorrectly here? The answer given in the book is $\dfrac{16}{105}$.
I didn't carefully check your setup, but your resulting integral is perfectly tractable. On the inside the $\theta$ part is constant, so you get an antiderivative of $r^3 \cos(\theta)/3$, which means the inner integral is $\sin(2 \theta)^3 \cos(\theta)/3$. This is then not so hard to integrate (use the double angle formula and then one of the standard tricks for integrals of the form $\int \sin(x)^m \cos(x)^n dx$).