I have to calculate $$\iint_{S} 2 x y \,d x \,d y$$ while $$S=\left\{(x, y) \in \mathbb{R}^{2} \mid 0 \leq x^{2}-y^{2} \leq 1,0 \leq x y \leq 1, y \geq 0\right\}$$
so I chose the change variables $$\vec{g}(x, y)=\left(x^{2}-y^{2}, x y\right)=(u, v)$$ thus I get the new domain is $$\vec{g}(S)=\left\{(u, v) \in \mathbb{R}^{2} \mid 0 \leq u \leq 1,0 \leq v \leq 1\right\}$$
the jacobian I got is $$J= \frac{1}{2\left(x^{2}+y^{2}\right)}$$ but now I am stuck with the expression of $$\iint\frac{xy}{x^2+y^2}\,du\,dv$$ what should I do next? for xy I can subtitute v, but what about $x^2+y^2$?
With your change of variables $\frac{xy}{x^2+y^2}=\frac{v}{\sqrt{u^2+4v^2}}$, so your integral is$$\begin{align}\int_0^1du\int_0^1\frac{vdv}{\sqrt{u^2+4v^2}}&=\frac18\int_0^1du[\sqrt{u^2+4}-u]\\&=\frac{1}{16}[u\sqrt{u^2+4}+4\operatorname{arsinh}\tfrac{u}{2}-u^2]_0^1\\&=\frac{\sqrt{5}+4\operatorname{arsinh}\tfrac12-1}{16}.\end{align}$$