Transform the given integral in Cartesian coordinates to polar coordinates and evaluate the polar integral:
(b) $\iint_D \sin(x^2+y^2)~dA$, where $D$ is the region in the first quadrant bounded by the lines $x=0$ and $y=\sqrt{3}\cdot x$ and the circles $x^2+y^2=2\pi$ and $x^2+y^2=3\pi$.
In Cartesian coordinates I know $0\le y\le \sqrt{3 \pi}, \space0\le x \le \frac{\sqrt{3\pi}}{4}$
I also know $x=r\cos\theta$, $y=r\sin\theta$
How should I proceed from here?
$r$ remains the same $r= \sqrt{3\pi}$? So $0\le r\le\sqrt{3\pi}$, $0\le\theta\le \frac{\pi}{2}$. Is this correct?
No, it is not correct. The polar representation you give yields a quarter circle of radius $\sqrt{3\pi}$ on the first quadrant, which is obviously not the region described by the question.
The first thing I always do to solve these types of problems is to sketch the region of integration. In this case, it turns out to be extremely helpful:
The region $D$ you are concerned about I highlighted in yellow. It is easy to see that $x^2+y^2=2\pi$ and $x^2+y^2=3\pi$ are just circles of radius $\sqrt{2\pi}$ and $\sqrt{3\pi}$ respectively. Similarly, it can be seen that $y=\sqrt{3}\cdot x$ and $x=0$ correspond to $\theta=60^{\circ}=\pi/3$ (Since $\arctan(\sqrt{3})=\pi/3$) and $\theta=90^{\circ}=\pi/2$ respectively. Hence, the region can be described in polar coordinates as: $$D=\{(r,\theta)\in \mathbb{R}^2\mid \sqrt{2\pi}\leq r\leq \sqrt{3\pi}, \pi/3\leq \theta\leq \pi/2\}$$ Hence, converting the double integral to polar coordinates gives (Don't forget the Jacobian): $$\iint_D \sin(x^2+y^2)~dA=\int_{\pi/3}^{\pi/2} \int_{\sqrt{2\pi}}^{\sqrt{3\pi}} \sin(r^2)\cdot r~dr~d\theta$$ This should be easy to evaluate.