Evaluate $\iint {(x+y)^2} dxdy$ over the region in the first quadrant bounded by $x^2-y^2=a$, $x^2-y^2=b$, $2xy=c$ and $2xy=d$ where $0<a<b$, $0<c<d$.
I took $x^2-y^2=u$ and $2xy=v$ and tried to express $(x+y)^2$ in terms of $u,v$. Then I changed the variables using the Jacobian from $x,y$ to $u,v$ with the limits. However after evaluating the integral the answer specified in the book doesn't match. The answer is $(b-a)(d-c)/4$
Either the original integrand or the provided solution is incorrect.
As you've shown, under your change of variables,
$$\begin{cases}u = x^2 - y^2 \\ v = 2xy\end{cases} \implies \begin{cases}x = \dfrac1{\sqrt2} \sqrt{u+\sqrt{u^2+v^2}} \\ y = \dfrac1{\sqrt2} \dfrac v{\sqrt{u+\sqrt{u^2+v^2}}}\end{cases} \implies dx\,dy = \frac{du\,dv}{4\sqrt{u^2+v^2}}$$
so the given integral (partially) evaluates to
$$\begin{align*} \iint (x+y)^2 \, dx \, dy &= \frac14 \int_c^d \int_a^b \frac{v + \sqrt{u^2+v^2}}{\sqrt{u^2+v^2}} \, du \, dv \\ &= \frac14 \int_c^d \left(b-a + v \log \frac{\sqrt{v^2+a^2}-a}{\sqrt{v^2+b^2}-b}\right) \, dv\\ &= \frac{(b-a)(d-c)}4 + \frac14 \int_c^d v \log \frac{\sqrt{v^2+a^2}-a}{\sqrt{v^2+b^2}-b} \, dv \end{align*}$$
and so on; you can confirm this result does not simplify to the expected answer.
On the other hand, notice that $x^2+y^2 = \sqrt{u^2+v^2}$; this is probably what the author meant for you to work with.