In my text given the integral $\int_{\partial{D}} xy\,dx$, and that
$$\int_{\partial{D}} xy\,dx=-\int\int_{D}x\,dx\,dy = - \int\int r\cos \theta\,r\,dr\,d\theta$$
I'm not really understanding the change to polar, and how we get this result.
I see that $x=r\cos\theta$ but how does $dx \to r\,dr$ and $dy \to d\theta$?
First equality holds via Green's theorem $$\int_{\partial D} xy\, dx = - \iint_{D} x d(x,y).$$
For the second one the substitution is made (polar coordinates) \begin{cases} x= r\cos\theta \\ y= r\sin \theta.\end{cases}
Thus we obtain
$$- \iint_{D} x d(x,y) = -\int \int r\cos \theta |J| dr d\theta,$$ where $J$ is a Jacobian determinant of the coordinate conversion formula $$J =\begin{vmatrix} \frac{\partial x}{\partial r} & \frac{\partial x}{\partial \theta} \\[8pt] \frac{\partial y}{\partial r} & \frac{\partial y}{\partial \theta} \end{vmatrix} =\begin{vmatrix} \cos\theta & -r\sin\theta \\ \sin\theta & r\cos\theta \end{vmatrix} =r\cos^2\theta + r\sin^2\theta = r.$$
Therefore, $$\int \int r\cos \theta |J| dr d\theta = \int \int r^2\cos \theta\, dr d\theta.$$
You should familiarise yourself with the formula for integration by substitution for multiple variables, here is the link.