Need to calculate $\int_{0}^{R}dx\int_{-\sqrt{{R}^{2}-{x}^{2}}}^{\sqrt{{R}^{2}-{x}^{2}}}cos({x}^{2}+{y}^{2})dy$
My steps:
Domain of integration is the circle with center (0,0) and radius R;
$x = \rho \cos \varphi ,\: y = \rho \sin \varphi,\: \rho \in \left[0,R \right],\: \varphi \in [0, 2\pi )$;
$\int_{0}^{R}dx\int_{-\sqrt{{R}^{2}-{x}^{2}}}^{\sqrt{{R}^{2}-{x}^{2}}}\cos\left( {x}^{2} + {y}^{2}\right) dy = \int_{0}^{R}d\rho \int_{0}^{2\pi }cos({\rho }^{2})d\varphi = 2\pi \int_{0}^{R}cos({\rho }^{2})d\rho$
As I know from WolframAlpha last integral can not be calculated using Elementary functions.
You took the wrong bounds for the angle, and forgot to insert the Jacobian determinant:
$$ \int_{0}^{R}\int_{-\sqrt{R^2-x^2}}^{\sqrt{R^2-x^2}}\cos(x^2+y^2)\,dy \,dx = \int_{-\pi/2}^{\pi/2}\int_{0}^{R}\rho \cos(\rho^2)\,d\rho\,d\theta = \frac{\pi}{2}\sin(R^2).$$