$$\int_{L}{{e^{-(x^2+y^2)}}(\cos2xydx+\sin2xydy)}$$, where L is this circle $x^2+y^2 = R^2$.
Since I have to use Green's formula, I computed $\frac{\partial{P}}{\partial{y}}$ and $\frac{\partial{Q}}{\partial{x}}$ and to my surprise they didn't cancel each other. Now I'm left with this $\iint_{D}{4y{e^{-(x^2+y^2)}}\cos2xy}dxdy$. I don't think this is easily computable. This was on my midterm a few hours ago and I got 4/5. I'm thinking the only issue has to be in this problem that I was unable to complete. I would really appreciate any help.Thanks
First of all, note that you can forget about the exponential, since it reduces to the constant $e^{-R^2}$. This is exactly what you are missing.
Indeed, apart from that multiplicative constant, the integral is $$I=\int_L \cos(2xy)\, dx + \sin(2xy)\, dy.$$ As you noticed, the differential form in the integrand is not closed (this is how to properly express that $\partial_y P = \partial_x Q$. The way you wrote it made no sense: who is $P$ and who is $Q$? You never defined them).
The fact that the form is not closed simply means that you can not immediately conclude that the integral vanishes.
However we can still use the formulas of Gauss--Green, as you noticed: $$ I=2\iint_{D_R}\left[ y\cos(2xy) +x\sin(2xy)\right]\, dxdy, $$ where $D_R$ denotes the disk of radius $R$. Now we see that the right-hand side indeed IS zero: use the change of variable $y\mapsto -y$ to prove it.
We conclude that the integral indeed was zero.