Let $D$ denotes area of the first quadrant which bounded by two lines $y=0, y=x$ and
two hyperbolas $xy=1, x^2-y^2=1$.
Evaluate double integral $$\iint_D \frac{x^4-y^4}{1+xy}dxdy$$
Since we can't integrate this neither $x$ nor $y$, should do $u, v$ substitution.
General problem which like this has similar bounds so I can find change of variables easily.
(For example :
If I have these bounds : $$x+y=1, x+y=3\\ x-y=1, x-y=3$$
I can find $x+y=u, \quad x-y=v$ easily. )
But the original problem is, there is no similar bounds So I don't know what should I $uv-$sub.
The only thing what can I find is given two hyperbolas are same but rotated $\frac{\pi}{4}$. (or $-\frac{\pi}{4}$).
Is there any hints?
If I do $$xy=u \\ x^2-y^2=v$$
then$$ dxdy=\frac{1}{2(x^2+y^2)}dudv$$
So the original double integral changes to $$\frac{1}{2}\iint_{D*} \frac{v}{1+u}dudv$$
But In this case, I can't find the bound of $u$ and $v$.
Let us consider your second approach: note that the map $$(u,v)=\Phi(x,y)=(xy,x^2-y^2)$$ transforms bijectively $$D=\{(x,y): 0\leq xy\leq 1, 0\leq x^2-y^2\leq 1,x\geq 0\}$$ into $D^*=[0,1]\times [0,1]$.
Therefore, since $dxdy=\frac{1}{2(x^2+y^2)}dudv$, it follows that $$\iint_{D} \frac{x^4-y^4}{1+xy}dxdy=\frac{1}{2}\iint_{[0,1]\times [0,1]} \frac{v}{1+u}dudv=\frac{1}{2}\left[\frac{v^2}{2}\right]_0^1\left[\ln(1+u)\right]_0^1=\frac{\ln(2)}{4}.$$