Let $S$ be the region in the first quadrant bounded by the curves $xy=1$, $xy=3$, $x^2-y^2=1$, and $x^2-y^2=4$. Compute $$\iint _S (y^2+x^2)\, dA.$$
I have tried to use the change of variables $u=xy$ and $v=x+y$. Then $$\begin{align}x^2+y^2=v^2-2u \\ x^2-y^2=v^2-2u \end{align}$$
Since $v^2=x^2-y^2+2u$, $$0 \le v^2 \le 4-1=3$$, so $$0 \le v \le \sqrt{3}$$
The integral becomes $$\begin{align} \iint _S (y^2+x^2)\, dA= \int_0^\sqrt{3} \int_1^3 (v^2-2u) \left|J \right| \,du\,dv \end{align}$$ but I can't solve $x,y$ in terms of $u,v$.
I have tried another one, $u=x^2+y^2$ and $v=x^2-y^2$.
I get $x=\sqrt{\frac{u+v}{2}}$ and $y=\sqrt{\frac{u-v}{2}}$
Then $1\le xy=\frac{\sqrt{u^2-v^2} }{2} \le 3$, and $$4\le u^2-v^2 \le 36$$
The integral becomes $$\begin{align} \iint _S (y^2+x^2)\, dA= \int_1^4 \int_{\sqrt{4+v^2}}^{\sqrt{36+v^2}} u \left|J \right| \,du\,dv \end{align}$$
Hope someone could give me some suggestion how to do it.
EDIT
The substitution $u=xy$, $v=x^2-y^2$ does the job. The Jacobian with respect to $x$ and $y$ is
$$ J=\left(\begin{matrix} y & x\\ 2x & -2y \end{matrix}\right) $$
so $|\det(J)|=2(x^2+y^2)$, which means $dudv=2(x^2+y^2)dxdy$ and your integral becomes $$\begin{align} \iint _S (y^2+x^2) dxdy= \int _1^3 \int_1^4 2 \;dudv \end{align}$$
As Ian said, this is no more than a special case of the "inverse method", used also in order to calculate one-dimensional integrals. E.g., if you have
$$ \int \tan(x)^4 dx $$
and you apply the substitution $t=\tan(x)$, then it is easy to obtain $dt=(1+\tan(x)^2)dx=(1+t^2)dx$, and then $dx=\frac{dt}{1+t^2}$ (of course, in this case it is equally easy to start from $x=\arctan(t)$ and so on, but sometimes you can avoid some annoying calculation).