Let $D$ be the region that's bounded by $xy=a, xy=b, y^2-x^2=1, y=x$ in the first quadrant. Calculate the integral $\iint_D(y^2-x^2)^{xy}(x^2+y^2)dxdy$.
Firstly, I was able to show that the boundary of this region is a Jordan curve and that $D$ is bounded. Therefore, I can use either Green's theorem (which I couldn't find a good use for here) or change of variables. So I tried using polar coordinates but that didn't get me anywhere. I also tried the substitution $u=y-x, v=y+x$ such that $x=-\frac{1}{2}u+\frac{1}{2}v, y=\frac{1}{2}u+\frac{1}{2}v$ such that $|J|=\frac{1}{4}$.
I had some trouble here showing what the range of integration is using $u,v$, so for now we'll call it $\Omega$. Therefore:
$\iint_D (y^2-x^2)^{xy}(x^2+y^2)dxdy=\iint_\Omega(uv)^{\frac{1}{4}v^2-\frac{1}{4}u^2}(\frac{1}{2}u^2+\frac{1}{2}v^2)dudv$.
My problems with this particular coordinate substitution are that it's difficult to calculate the range of integration and that the integral itself (regardless of the range) isn't very friendly. I'd like to know if there's some other direction that should be pursued or if there's an easier coordinate substitution.
***Edit: I do believe that the best way to do this is to use the substitution $u=xy, v=y^2-x^2$ because then $a<u<b, 0<v<1$, but I have trouble calculating this substitution's Jacobian. This is due to the fact that it's not easy to write $x,y$ as functions of $u,v$, so if anyone can help with this direction that would be appreciated.
This is to complete amd's comment. Compute the Jacobian of $u,v$ with respect to $x,y$: $$ \frac{\partial(u,v)}{\partial(x,y)}=\begin{vmatrix} x & y \\ 2\,x & -2\,y\end{vmatrix}=-2(x^2+y^2)\implies \frac{\partial(x,y)}{\partial(u,v)}=-\frac{1}{2(x^2+y^2)}. $$ Now, when you do the change of variables, the Jacobian cancels the $x^2+y^2$ term.