Question:
Use the change of variables $x = \cos\theta$ and $y=\sin\theta$ to find $$\int_{0}^{2} \int_{0}^{\sqrt{2x-x^2}} \sqrt{x^2 + y^2} \,dy\,dx$$
Attempt 1: Assuming coordinates are supposed to be $\,dr\,d\theta$.
As $r=1$ (constant), the Jacobian determinant turns the integral to $$\iint0\,dr\,d\theta$$ Is this the way to go?
Attempt 2: Differentiating x and y with respect to $\theta$, $$dx=-\sin\theta d\theta$$ $$dy=\cos\theta d\theta$$
But when trying to evaluate limits for $x$ we get $$2=\cos\theta$$ But this can't be possible??
Also does an integral increment of $d\theta^2$ make sense?
I think you'd better again read your textbook with more effort, you apparently misled the concept of changing variables.
The integration domain is $$ \bigg\{{(x,\,y) | 0\leq x\leq 2,\;0\leq y\leq \sqrt{2x-x^2}}\bigg\}, $$ which is the upper semi-disc enclosed by the circle $(x-1)^2 + y^2 = 1$.
By letting $$ \begin{cases} x = r\cos\theta \\ y=r\sin\theta, \end{cases} $$ since $0\leq x\leq 2$ and $y\geq 0$, it follows that $0\leq\theta\leq\frac{\pi}{2}$, also, since $y\leq\sqrt{2x-x^2}$, it follows that $r\leq 2\cos\theta$.
Therefore, by changing the variables, the original integral becomes $$ \int_{0}^{\frac{\pi}{2}}\int_{0}^{2\cos\theta}r^2\,dr\,d\theta. $$
Note: In your way of changing variables, you should note that $r$ is not a constant. You are not integrating over a circle but a semi-disc.